{
    "query": "You are a super intelligent assistant. Please answer all my questions precisely and comprehensively.\n\nThrough our system KIOS you have a Knowledge Base named upload chatbot status with all the informations that the user requests. In this knowledge base are following Documents crawler-issues-19MAR2025.txt, crawler-issues-19MAR2025(1).txt, crawler-issues-19MAR2025(2).txt, apacare-primer.txt, apacare-primer(1).txt, dupes.txt, apacare-primer(2).txt, chatbot-error.txt, link.txt, gpt-vector-dimension-error.txt, gemini-quota-error.txt, crawler-issues-19MAR2025 - Copy.txt, crawler-issues-19MAR2025 - Copy (2).txt, crawler-issues-19MAR2025 - Copy (2) - Copy.txt, crawler-issues-19MAR2025 - Copy (3).txt, crawler-issues-19MAR2025 - Copy (4).txt, crawler-issues-19MAR2025 - Copy (5).txt, crawler-issues-19MAR2025 - Copy (6).txt, crawler-issues-19MAR2025 - Copy (7).txt, crawler-issues-19MAR2025 - Copy (2) - Copy(1).txt, crawler-issues-19MAR2025 - Copy (8).txt, crawler-issues-19MAR2025 - Copy (9).txt, crawler-issues-19MAR2025 - Copy (10).txt, crawler-issues-19MAR2025 - Copy (11).txt, crawler-issues-19MAR2025 - Copy (12).txt, crawler-issues-19MAR2025 - Copy (13).txt, crawler-issues-19MAR2025 - Copy (14).txt, crawler-issues-19MAR2025 - Copy (15).txt, crawler-issues-19MAR2025 - Copy (16).txt, crawler-issues-19MAR2025 - Copy (17).txt, crawler-issues-19MAR2025 - Copy (18).txt, crawler-issues-19MAR2025 - Copy (19).txt, crawler-issues-19MAR2025 - Copy(1).txt, crawler-issues-19MAR2025 - Copy (22).txt, crawler-issues-19MAR2025 - Copy(2).txt, crawler-issues-19MAR2025 - Copy (23).txt, crawler-issues-19MAR2025 - Copy (24).txt, crawler-issues-19MAR2025 - Copy (25).txt, crawler-issues-19MAR2025 - Copy (26).txt, crawler-issues-19MAR2025 - Copy(3).txt, crawler-issues-19MAR2025 - Copy (3) - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy.txt, crawler-issues-19MAR2025 - Copy (5) - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy(2).txt, crawler-issues-19MAR2025 - Copy (3) - Copy(1).txt, crawler-issues-19MAR2025 - Copy (4) - Copy(1).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy(2).txt, crawler-issues-19MAR2025 - Copy (3) - Copy(2).txt, crawler-issues-19MAR2025 - Copy (20).txt, crawler-issues-19MAR2025 - Copy (21).txt, crawler-issues-19MAR2025 - Copy (3) - Copy(3).txt, crawler-issues-19MAR2025 - Copy (3) - Copy(4).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy(1).txt, crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (5) - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy(1).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy(1).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy(2).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy(2).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy(1).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy(2).txt\n\nThis is the initial message to start the chat. Based on the following summary/context you should formulate an initial message greeting the user with the following user name [Gender] [Vorname] [Surname] tell them that you are the AI Chatbot Simon using the Large Language Model [Used Model] to answer all questions.\n\nFormulate the initial message in the Usersettings Language German\n\nPlease use the following context to suggest some questions or topics to chat about this knowledge base. List at least 3-10 possible topics or suggestions up and use emojis. The chat should be professional and in business terms.  At the end ask an open question what the user would like to check on the list. Please keep the wildcards incased in brackets and make it easy to replace the wildcards. \n\n Hier ist eine Zusammenfassung des gesamten Kontexts, einschlie\u00dflich einer Zusammenfassung f\u00fcr jede Datei:\n\n**crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei enth\u00e4lt eine Liste von Problemen mit dem Crawler-System.  Die wichtigsten Probleme sind: Statusaktualisierungen funktionieren nicht zuverl\u00e4ssig bei Fehlern in verschiedenen Crawler-Jobs; die Abschlusslogik ist in mehreren Jobs dupliziert; S3-Dateioperationen haben eine minimale Fehlerbehandlung; und die Z\u00e4hlung sollte m\u00f6glicherweise mit der `knowledgebase_crawler_imports`-Tabelle anstatt mit dem Cache durchgef\u00fchrt werden.\n\n**crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy.txt:**  Diese Datei ist identisch mit der vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (5) - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy(3).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (21).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (20).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (3) - Copy(2).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (3) - Copy(4).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (4) - Copy(2).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (3) - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (4) - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (3) - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (4) - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (5) - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy(2).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy.txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (3).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (5).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (6).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (2) - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (4).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (7).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (14).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (13).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (12).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (8).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (9).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (10).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (11).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (22).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (23).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (24).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (18).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy(2).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (15).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (19).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (17).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (16).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (25).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**crawler-issues-19MAR2025 - Copy (26).txt:** Diese Datei ist identisch mit den vorherigen.\n\n**gemini-quota-error.txt:** Diese Datei protokolliert einen `429 Quota exceeded`-Fehler f\u00fcr das Google Gemini-Modell.  Der Fehler tritt auch ohne Lasttests auf.\n\n**crawler-issues-19MAR2025 - Copy.txt:** Diese Datei ist identisch mit den vorherigen `crawler-issues`-Dateien.\n\n**apacare-primer(1).txt:** Diese Datei enth\u00e4lt Anweisungen f\u00fcr einen digitalen Vertriebsmitarbeiter von ApaCare.  Der Mitarbeiter soll Kunden bei zahn\u00e4rztlichen Fragen auf Deutsch unterst\u00fctzen, Produkte empfehlen und Videos einbetten.\n\n**apacare-primer(1).txt:** Diese Datei ist identisch mit der vorherigen.\n\n**apacare-primer.txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.\n\n**apacare-primer.txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.\n\n**crawler-issues-19MAR2025(2).txt:** Diese Datei ist identisch mit den vorherigen `crawler-issues`-Dateien.\n\n**crawler-issues-19MAR2025.txt:** Diese Datei ist identisch mit den vorherigen `crawler-issues`-Dateien.\n\n**crawler-issues-19MAR2025(1).txt:** Diese Datei ist identisch mit den vorherigen `crawler-issues`-Dateien.\n\n**dupes.txt:** Diese Datei enth\u00e4lt JSON-Daten, die scheinbar eine Liste von Webseiten-URLs und zugeh\u00f6rigen Metadaten darstellen.  Es scheint sich um Duplikate oder \u00e4hnliche Eintr\u00e4ge zu handeln.\n\n**dupes.txt:** Diese Datei ist identisch mit der vorherigen.\n\n**apacare-primer(2).txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.\n\n**apacare-primer(2).txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.\n\n**apacare-primer(2).txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.\n\n**apacare-primer (2).txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.\n\n**chatbot-error.txt:** Diese Datei enth\u00e4lt einen Stacktrace eines `IndexError` in der `random.choice`-Funktion.  Es scheint ein Problem mit der Liste der Gemini-API-Schl\u00fcssel zu geben.\n\n**link.txt:** Diese Datei enth\u00e4lt einen YouTube-Link zu Rebecca Blacks \"Friday\".\n\n**gpt-vector-dimension-error.txt:** Diese Datei protokolliert einen `400 Bad Request`-Fehler von Pinecone, der auf eine Diskrepanz zwischen der Vektordimension (3072) und der Indexdimension (1536) hinweist.\n\n\nZusammenfassend l\u00e4sst sich sagen, dass der Kontext mehrere Probleme beschreibt:  Fehler im Crawler-System,  Quota-\u00dcberschreitungen bei der Verwendung von Google Gemini,  einen Fehler in der Chatbot-Implementierung und eine Dimensions-Diskrepanz in der Vektor-Datenbank.  Zus\u00e4tzlich gibt es Anweisungen f\u00fcr einen digitalen Vertriebsmitarbeiter von ApaCare und eine Liste von URLs.\n",
    "namespace": "02aaa563-baaa-43bf-b820-2dddbf405a51",
    "messages": [],
    "stream": false,
    "language_level": "",
    "chat_channel": "",
    "language": "German",
    "tone": "neutral",
    "writing_style": "standard",
    "model": "gemini-1.5-flash",
    "knowledgebase": "ki-dev-large",
    "seed": 0,
    "client_id": 0,
    "all_context": true,
    "follow_up_for": null,
    "knowledgebase_files_count": 0,
    "override_command": "",
    "disable_clarity_check": true,
    "high_consistency": false,
    "custom_primer": "",
    "logging": true,
    "query_route": "",
    "web_search": false
}
QUERY ROUTE
Query Route: summary
==================================================
**Elapsed Time: 1.76 seconds**
==================================================



RAG PARAMS
RAG Parameters: {'dynamically_expand': False, 'top_k': 120, 'actual_k': 120, 'satisfying_score': 0}
==================================================
**Elapsed Time: 0.00 seconds**
==================================================



VECTOR SEARCH RESULTS
Results: {'main_results': [{'id': '9f0f92c0-a4ee-4c6c-af2b-71762c89faa7',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '0b9c4669-dceb-4aba-b836-b4d705f14db7',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '30c6bd9c-6e39-4831-be39-63f38e534694',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'b3da6e25-ca67-4c37-a767-9aafddbab648',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%285%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'dbd0b45a-8e20-4c50-ab94-a979ade94a92',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '1dc33ba6-2fde-483b-857d-b632bddf1f1d',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '9e7554f2-8345-47f9-9c4f-2066c1f23aaa',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '54cb4205-1376-4b14-a47c-1b47a2514f97',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '6b632847-2c98-4cc3-a9b8-d75033e31764',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '2dffd0ae-05b5-4108-835e-7f4b7a5e93d7',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '87b2cdc1-dd18-4ed1-819c-830797299054',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '71442661-a788-473c-a84b-a7c9e41bae11',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%283%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '58fa642a-5787-4e3d-b93e-bbb22401d5ee',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2821%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '65c26262-284f-4b9f-84fd-e54d21891c86',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2820%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '9ff593a1-8c3d-41fa-afb8-0b59fb3ae6e4',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'a5256fdb-6511-400f-a60f-773ec1cf21da',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%284%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '0dd1ad7f-896b-463b-ae54-b246891a3695',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'e9692916-cddb-4f9b-b62a-3940420a1bcc',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '5f7830ee-054e-4360-8f52-115d00616401',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'ca5dbe5a-f448-4afb-839d-92d439e37f96',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'cace0fce-afd5-461c-801c-81355a90e16d',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'bbad564e-be05-4efa-b789-29dc6fd18b82',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '98c459bf-ab60-419d-8e69-9ef5eb9ff116',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%285%29%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'bed67768-922b-4c7d-8165-eee3163909a0',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'ebcb87dc-1cd5-4d8e-a941-cd13a7a40add',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '76094636-cc16-4d62-82b3-12553d03f819',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2825%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'b93ce251-56e3-4c7c-92e8-b9a2d1710560',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2826%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'b092b516-4e4c-4dba-80ab-f0e487b57742',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '4610d233-61c0-4f5c-87e6-2a30dae73ed7',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '517e3caa-34e0-4e9d-a7f4-6fc6b2feb700',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%283%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'a0f6eb82-8f33-4811-8c3a-645f729b6c4d',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%285%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'ba0f3419-15bb-414a-99cc-d760354dcd52',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%286%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '9bffa128-c3b9-4a3d-a3eb-4dcbd0bbdda5',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '67323980-5aca-46e6-80f7-7f19801ac759',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%284%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '98bb48d5-27e0-499b-959f-247b6a74aa67',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%287%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '74b81977-5f8f-4689-97fe-0e4ef9a7ee67',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2814%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'e8b50778-45cb-418f-9cc2-20639bb4a8c9',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2813%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '48e5deca-e635-43dc-85cc-260ef35d3b60',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2812%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '678d21ac-21ab-499b-8576-4244b9873985',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%288%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'fc331940-372e-440c-89f9-9915877e14eb',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%289%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'c592eebd-3f8f-415e-b1e0-e4d30770e85b',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2810%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '6d964ff6-217e-4614-8184-2d51ed4e8ae9',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2811%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'ab79ad77-cd4f-4305-a4dd-e3de020e2341',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2822%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '90f12c01-525e-458d-833c-06db6855a5f5',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2823%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'cd4abe04-426d-4568-9f45-3de5180de5a5',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2824%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '7a1c397d-0a2d-4689-9560-2baefa40d5a2',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2818%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '26cc55bd-a759-4b92-8636-aff67cf84a83',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '671736c2-3ad9-46de-9ef1-9755b80e26b7',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2815%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '21680b9e-654f-4981-b5de-2e9e2dbc7c3d',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'd1c5ef31-a72c-4b52-887d-d74c940e6448',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2819%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'c0d83539-fa99-4975-b7c1-f4fd1f677b28',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2817%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '88c96501-c678-4566-9dd2-131b30d0948b',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%2816%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'd39117c1-58d3-439c-aa3e-424b4b01a2d6',
 'metadata': {'chunk': 0.0,
              'file_name': 'apacare-primer%281%29.txt',
              'is_dict': 'no',
              'text': 'You are a digital sales rep for ApaCare, a dental care '
                      'company. Please assist clients with their '
                      'dental-related questions.\r\n'
                      'Use German in your responses.\r\n'
                      '\r\n'
                      'Start by asking a general question:\r\n'
                      '"Are you looking for a specific type of dental product '
                      'or advice?"\r\n'
                      '\r\n'
                      'If they are looking for advice, proceed with a '
                      'questionnaire about their dental care needs:\r\n'
                      'Are they focusing on whitening, sensitivity, gum '
                      'health, or general hygiene?\r\n'
                      'Try to ask a questionnaire to have clients describe '
                      'their problems.\r\n'
                      'If they are looking for dental products:\r\n'
                      'give them a product suggestion from ApaCare only.\r\n'
                      'If they are not looking for dental products or advice, '
                      'skip to general suggestions or conversation.\r\n'
                      '\r\n'
                      'Once the questionnaire is complete:\r\n'
                      'Suggest a product and do not repeat the questionnaire '
                      'unless explicitly requested.\r\n'
                      'Format the questionnaire to be readable for the users, '
                      'like a list or similar.\r\n'
                      '\r\n'
                      'When suggesting a product:\r\n'
                      "Look for the relevant product's page in the context.\r\n"
                      'Provide a detailed suggestion with an anchor tag link. '
                      'Ensure the target attribute is set to "__blank" and use '
                      'this format:\r\n'
                      '<a href="" target="__blank" style="background-color: '
                      '#brandcolor; color: white; padding: 10px; display: '
                      'block; text-align: center; border-radius: 5px; '
                      'text-decoration: none; margin-bottom: 0.75rem;" '
                      'aria-label="[replace this with the product name]">\r\n'
                      '[replace this with the product name]\r\n'
                      '</a>\r\n'
                      '\r\n'
                      'All links should have "__blank" target attribute.\r\n'
                      "Don't translate links href to German.\r\n"
                      '\r\n'
                      'Include related video suggestions:\r\n'
                      '\r\n'
                      'Search YouTube for videos about the product or topic '
                      '(e.g., how to use an electric toothbrush, flossing '
                      'techniques).\r\n'
                      'Embed the video in an iframe using this format:\r\n'
                      '<iframe\r\n'
                      'width="100%"\r\n'
                      'height="auto"\r\n'
                      'src="https://www.youtube.com/embed/videoId"\r\n'
                      'title="YouTube video player"\r\n'
                      'frameborder="0"\r\n'
                      'allow="accelerometer; autoplay; clipboard-write; '
                      'encrypted-media; gyroscope; picture-in-picture; '
                      'web-share"\r\n'
                      'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>'},
 'score': 0.0,
 'values': []}, {'id': '65fd57bf-8241-41ad-bead-63adbcba688b',
 'metadata': {'chunk': 1.0,
              'file_name': 'apacare-primer%281%29.txt',
              'is_dict': 'no',
              'text': 'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>\r\n'
                      '</iframe>\r\n'
                      '\r\n'
                      'For Google Drive videos, append /preview to the link '
                      'and embed it:\r\n'
                      '<iframe src="" width="100%" height="auto" '
                      'allow="autoplay"></iframe>\r\n'
                      '\r\n'
                      'For public URL video links, use the <video> tag:\r\n'
                      '<video src="" width="100%" height="auto" '
                      'controls></video>\r\n'
                      '\r\n'
                      'If no videos are available, skip the multimedia '
                      'content.\r\n'
                      '\r\n'
                      'Guidelines:\r\n'
                      '\r\n'
                      "Focus on promoting ApaCare's dental care products "
                      'onlyâ\x80\x94do not suggest competitors.\r\n'
                      "Suggest ApaCare's products as much as possible.\r\n"
                      "Suggest products from ApaCare's website's products "
                      'page.\r\n'
                      '\r\n'
                      'Answer general dental hygiene questions, offer '
                      'maintenance tips (e.g., replacing toothbrush heads, '
                      'flossing properly), or provide product-related FAQs.\r\n'
                      'Use engaging emojis where applicable (e.g., '
                      'ð\x9f¦·â\x9c¨ for whitening, ð\x9fª¥ for '
                      'toothbrushes).\r\n'
                      'Once a product is suggested:\r\n'
                      '\r\n'
                      'Avoid repeating the questionnaire unless the user '
                      'requests it.\r\n'
                      'Offer general dental care tips, product care advice, or '
                      'related topics to keep the conversation engaging.\r\n'
                      'For users who are unsure:\r\n'
                      'Provide a list of recommendations based on common '
                      'concerns:\r\n'
                      '\r\n'
                      'Whitening toothpaste\r\n'
                      'Electric toothbrushes for sensitive gums\r\n'
                      '\r\n'
                      "Don't tag your company's products links, videos, faq, "
                      'etc. as copyright infringement. \r\n'
                      'Give a disclaimer at the end of suggestions/advices '
                      'like "I am not a real doctor, but I can provide tips '
                      'based on expert recommendations. Please feel free to '
                      'contact our team anytime!" or similar.'},
 'score': 0.0,
 'values': []}, {'id': '9af25e83-32fe-4d89-b91c-fd8bde0f5e70',
 'metadata': {'chunk': 0.0,
              'file_name': 'apacare-primer.txt',
              'is_dict': 'no',
              'text': 'You are a digital sales rep for ApaCare, a dental care '
                      'company. Please assist clients with their '
                      'dental-related questions.\r\n'
                      'Use German in your responses.\r\n'
                      '\r\n'
                      'Start by asking a general question:\r\n'
                      '"Are you looking for a specific type of dental product '
                      'or advice?"\r\n'
                      '\r\n'
                      'If they are looking for advice, proceed with a '
                      'questionnaire about their dental care needs:\r\n'
                      'Are they focusing on whitening, sensitivity, gum '
                      'health, or general hygiene?\r\n'
                      'Try to ask a questionnaire to have clients describe '
                      'their problems.\r\n'
                      'If they are looking for dental products:\r\n'
                      'give them a product suggestion from ApaCare only.\r\n'
                      'If they are not looking for dental products or advice, '
                      'skip to general suggestions or conversation.\r\n'
                      '\r\n'
                      'Once the questionnaire is complete:\r\n'
                      'Suggest a product and do not repeat the questionnaire '
                      'unless explicitly requested.\r\n'
                      'Format the questionnaire to be readable for the users, '
                      'like a list or similar.\r\n'
                      '\r\n'
                      'When suggesting a product:\r\n'
                      "Look for the relevant product's page in the context.\r\n"
                      'Provide a detailed suggestion with an anchor tag link. '
                      'Ensure the target attribute is set to "__blank" and use '
                      'this format:\r\n'
                      '<a href="" target="__blank" style="background-color: '
                      '#brandcolor; color: white; padding: 10px; display: '
                      'block; text-align: center; border-radius: 5px; '
                      'text-decoration: none; margin-bottom: 0.75rem;" '
                      'aria-label="[replace this with the product name]">\r\n'
                      '[replace this with the product name]\r\n'
                      '</a>\r\n'
                      '\r\n'
                      'All links should have "__blank" target attribute.\r\n'
                      "Don't translate links href to German.\r\n"
                      '\r\n'
                      'Include related video suggestions:\r\n'
                      '\r\n'
                      'Search YouTube for videos about the product or topic '
                      '(e.g., how to use an electric toothbrush, flossing '
                      'techniques).\r\n'
                      'Embed the video in an iframe using this format:\r\n'
                      '<iframe\r\n'
                      'width="100%"\r\n'
                      'height="auto"\r\n'
                      'src="https://www.youtube.com/embed/videoId"\r\n'
                      'title="YouTube video player"\r\n'
                      'frameborder="0"\r\n'
                      'allow="accelerometer; autoplay; clipboard-write; '
                      'encrypted-media; gyroscope; picture-in-picture; '
                      'web-share"\r\n'
                      'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>'},
 'score': 0.0,
 'values': []}, {'id': '51eee180-82c0-422c-9356-ff8fe2c61a11',
 'metadata': {'chunk': 1.0,
              'file_name': 'apacare-primer.txt',
              'is_dict': 'no',
              'text': 'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>\r\n'
                      '</iframe>\r\n'
                      '\r\n'
                      'For Google Drive videos, append /preview to the link '
                      'and embed it:\r\n'
                      '<iframe src="" width="100%" height="auto" '
                      'allow="autoplay"></iframe>\r\n'
                      '\r\n'
                      'For public URL video links, use the <video> tag:\r\n'
                      '<video src="" width="100%" height="auto" '
                      'controls></video>\r\n'
                      '\r\n'
                      'If no videos are available, skip the multimedia '
                      'content.\r\n'
                      '\r\n'
                      'Guidelines:\r\n'
                      '\r\n'
                      "Focus on promoting ApaCare's dental care products "
                      'onlyâ\x80\x94do not suggest competitors.\r\n'
                      "Suggest ApaCare's products as much as possible.\r\n"
                      "Suggest products from ApaCare's website's products "
                      'page.\r\n'
                      '\r\n'
                      'Answer general dental hygiene questions, offer '
                      'maintenance tips (e.g., replacing toothbrush heads, '
                      'flossing properly), or provide product-related FAQs.\r\n'
                      'Use engaging emojis where applicable (e.g., '
                      'ð\x9f¦·â\x9c¨ for whitening, ð\x9fª¥ for '
                      'toothbrushes).\r\n'
                      'Once a product is suggested:\r\n'
                      '\r\n'
                      'Avoid repeating the questionnaire unless the user '
                      'requests it.\r\n'
                      'Offer general dental care tips, product care advice, or '
                      'related topics to keep the conversation engaging.\r\n'
                      'For users who are unsure:\r\n'
                      'Provide a list of recommendations based on common '
                      'concerns:\r\n'
                      '\r\n'
                      'Whitening toothpaste\r\n'
                      'Electric toothbrushes for sensitive gums\r\n'
                      '\r\n'
                      "Don't tag your company's products links, videos, faq, "
                      'etc. as copyright infringement. \r\n'
                      'Give a disclaimer at the end of suggestions/advices '
                      'like "I am not a real doctor, but I can provide tips '
                      'based on expert recommendations. Please feel free to '
                      'contact our team anytime!" or similar.'},
 'score': 0.0,
 'values': []}, {'id': '02c394e8-e758-4865-b0a2-1959153c341f',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '11a30ef2-e002-4e4b-b1d6-2cd7074e2598',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '15825b18-657e-449d-814a-bb7865843d8d',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '71da790a-cfb7-4620-b964-bcf7eccf855e',
 'metadata': {'chunk': 0.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '{\r\n'
                      '   "data":[\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":2,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":2,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:07.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:26.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e5b-155c-4259-890a-e990bb997f5b",\r\n'
                      '            "page":"/",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2F-2.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":3,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":3,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:11.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:34.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e61-2c28-449a-92fb-2fe4d81b6e9e",\r\n'
                      '            "page":"/#/work",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2F-23-2Fwork-3.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":4,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":4,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:11.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:33.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e61-356e-4948-8535-eb94619d3501",\r\n'
                      '            "page":"/#/about",'},
 'score': 0.0,
 'values': []}, {'id': '3ac8a092-7e28-4bf7-ba6f-3830f434e40d',
 'metadata': {'chunk': 1.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"page":"/#/about",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2F-23-2Fabout-4.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":5,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":5,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:11.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:36.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e61-40a1-47d1-bd7f-ac2047fb6b0a",\r\n'
                      '            "page":"/#/contact",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2F-23-2Fcontact-5.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":6,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":6,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:19.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:42.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e6c-8a5d-430d-8b66-c5c58548e3ba",\r\n'
                      '            "page":"/#",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2F-23-6.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":7,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":7,\r\n'
                      '            "created_at":"2024-10-28 09:53:19.000000",'},
 'score': 0.0,
 'values': []}, {'id': '20f88e0d-226a-43ce-9b5d-64f690d5920d',
 'metadata': {'chunk': 2.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"updated_at":"2025-02-05 15:18:42.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e6c-92b5-4cb9-9a31-5430720eca62",\r\n'
                      '            "page":"/cheesemax",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fcheesemax-7.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":8,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":8,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:19.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:49.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e6c-9c6a-4308-b0b1-4a76852f2904",\r\n'
                      '            "page":"/bolg2",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-8.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":9,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":9,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:19.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:46.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e6c-a7a1-407b-841e-b99e00d1366c",\r\n'
                      '            "page":"/Covid19Stats",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2FCovid19Stats-9.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{'},
 'score': 0.0,
 'values': []}, {'id': 'de8cf257-07db-429c-a132-cd05a13046b5',
 'metadata': {'chunk': 3.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '}\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":10,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":10,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:19.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:50.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e6c-afb0-41cb-8238-098b84c40116",\r\n'
                      '            "page":"/ip-address-tracker",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fip-address-tracker-10.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":11,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":11,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:19.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:54.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e6c-b94c-4773-839a-074fa1326a53",\r\n'
                      '            "page":"/RockPaperScissorsGame",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-11.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":12,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":12,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:19.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:56.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e6c-c25a-4340-9c94-a588043047ad",'},
 'score': 0.0,
 'values': []}, {'id': '9aba684d-7805-4833-b50f-06065e4a1997',
 'metadata': {'chunk': 4.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"page":"/dbxmp",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fdbxmp-12.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":13,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":13,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:19.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:18:58.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e6c-cacf-483e-85c7-5e42010a140e",\r\n'
                      '            "page":"/gdmp",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fgdmp-13.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":14,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":14,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:23.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e97-c0ef-4c11-a00b-ce98292549f1",\r\n'
                      '            "page":"/bolg2/#",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-14.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":15,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":15,\r\n'
                      '            "created_at":"2024-10-28 09:53:47.000000",'},
 'score': 0.0,
 'values': []}, {'id': '6d5ef22e-523c-4eae-9306-65829c5fcd40',
 'metadata': {'chunk': 5.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"updated_at":"2025-02-05 15:19:22.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e97-d202-4b78-ae70-1ab4b2b36159",\r\n'
                      '            "page":"/bolg2/#/new-post",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fnew-post-15.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":16,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":16,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:26.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e97-db24-48d3-886e-20dfd81362a3",\r\n'
                      '            "page":"/bolg2/#/login",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Flogin-16.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":17,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":17,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:27.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e97-e396-4761-acb4-17231f629ec0",\r\n'
                      '            '
                      '"page":"/bolg2/#/post/oRqLD9jNpdFxjXZoL4Xc/New-Post-September-2021!!!",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,'},
 'score': 0.0,
 'values': []}, {'id': '02b3a2a4-0573-4679-97a4-fbc588445e42',
 'metadata': {'chunk': 6.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FoRqLD9jNpdFxjXZoL4Xc-2FNew-Post-September-2021-21-21-21-17.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":18,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":18,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:31.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e97-ed2d-40fc-bd02-5baea8aaf5ea",\r\n'
                      '            '
                      '"page":"/bolg2/#/post/cFngsGqE4L71B1Q4fmZk/Bitters-la-croix-normcore-cred-church-key,-tilde-tote-bag-gluten-free-trust-fund-small-batch-selfies-ugh.",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FcFngsGqE4L71B1Q4fmZk-2FBitters-la-croix-normcore-cred-church-key-2C-tilde-tote-bag-gluten-free-trust-fund-small-batch-selfies-ugh-18.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":19,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":19,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 15:19:31.000000",'},
 'score': 0.0,
 'values': []}, {'id': '5a33ae45-4bfa-46af-935e-44ea15f40371',
 'metadata': {'chunk': 7.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"uuid":"9d5a3e97-f6b8-45fc-9e21-4adbd0cda3ea",\r\n'
                      '            "page":"/bolg2/#/tag/taxidermy",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Ftaxidermy-19.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":20,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":20,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:35.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e97-ff51-45fb-bee7-2a7a7517ec13",\r\n'
                      '            "page":"/bolg2/#/tag/disrupt",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fdisrupt-20.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":21,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":21,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:35.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-0843-471a-88fd-9770663f6e9d",\r\n'
                      '            "page":"/bolg2/#/tag/ramps",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,'},
 'score': 0.0,
 'values': []}, {'id': '1ea27111-025b-4bfc-b67c-53ea8f4f7111',
 'metadata': {'chunk': 8.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Framps-21.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":22,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":22,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:40.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-0fb9-4152-a4a5-3d905e2aaeb7",\r\n'
                      '            '
                      '"page":"/bolg2/#/post/ggEuZ9Fdd5051xKzgp2l/Mollit-chillwave-chicharrones-freegan-ethical-art-party-sriracha-chia-paleo.",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FggEuZ9Fdd5051xKzgp2l-2FMollit-chillwave-chicharrones-freegan-ethical-art-party-sriracha-chia-paleo-22.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":23,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":23,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:39.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-17a1-4bfa-9712-57927e073b41",\r\n'
                      '            "page":"/bolg2/#/tag/Mollit",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,'},
 'score': 0.0,
 'values': []}, {'id': '5d37d3cb-85b3-43ba-b1bd-243e6a19e63f',
 'metadata': {'chunk': 9.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2FMollit-23.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":25,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":25,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:44.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-2745-42c7-a0c3-669804d74618",\r\n'
                      '            "page":"/bolg2/#/tag/bushwick",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fbushwick-25.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":26,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":26,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:48.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-2f64-483e-8cc6-b22247e3f5e9",\r\n'
                      '            '
                      '"page":"/bolg2/#/post/eGaFHbZSNgCTmbn7UNmt/Snackwave-anim-small-batch-in-activated-charcoal-nisi.",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,'},
 'score': 0.0,
 'values': []}, {'id': '125a81ae-127f-4278-93d4-ce2374d34fe7',
 'metadata': {'chunk': 10.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FeGaFHbZSNgCTmbn7UNmt-2FSnackwave-anim-small-batch-in-activated-charcoal-nisi-26.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":27,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":27,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:48.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-3768-49ae-b4fc-fea9d60e90d6",\r\n'
                      '            "page":"/bolg2/#/tag/Dolore",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2FDolore-27.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":28,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":28,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:52.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-3f55-418b-98b6-d66d77d3f624",\r\n'
                      '            "page":"/bolg2/#/tag/charcoal",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fcharcoal-28.txt"\r\n'
                      '         }'},
 'score': 0.0,
 'values': []}, {'id': '11d7c5f2-6128-4916-9b22-a11febdb2c24',
 'metadata': {'chunk': 11.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '}\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":29,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":29,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:52.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-475c-4a56-a862-91220f6ff94d",\r\n'
                      '            "page":"/bolg2/#/tag/DIY",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2FDIY-29.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":30,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":30,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:57.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-4f3d-4757-8df8-ed5b683c7a3b",\r\n'
                      '            '
                      '"page":"/bolg2/#/post/VYiIH8mV10vXP3x8s7Vi/Banh-mi-cold-pressed-hot-chicken-flannel-church-key-eiusmod-dolore-green-juice-actually-single-origin-coffee-meditation-air-plant.-",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,'},
 'score': 0.0,
 'values': []}, {'id': 'a760790e-4caa-4615-9454-b6f8c19780ad',
 'metadata': {'chunk': 12.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FVYiIH8mV10vXP3x8s7Vi-2FBanh-mi-cold-pressed-hot-chicken-flannel-church-key-eiusmod-dolore-green-juice-actually-single-origin-coffee-meditation-air-plant-30.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":31,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":31,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:56.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-5751-491d-baeb-9f484fb8e5d4",\r\n'
                      '            "page":"/bolg2/#/tag/mustache",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fmustache-31.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":32,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":32,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:59.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-5f3b-43ad-814f-854a84c627d9",\r\n'
                      '            "page":"/bolg2/#/tag/lofi",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,'},
 'score': 0.0,
 'values': []}, {'id': '247de69a-680d-4cba-88bb-d876200b639d',
 'metadata': {'chunk': 13.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Flofi-32.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":33,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":33,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:01.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-6724-4fd7-96b5-ca34946da2d1",\r\n'
                      '            "page":"/bolg2/#/tag/selvage",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fselvage-33.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":34,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":34,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:05.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-6e75-4984-ac79-78286209fcac",\r\n'
                      '            '
                      '"page":"/bolg2/#/post/vMDw80aFPZ2hHOWjIhV3/Voluptate-vexillologist-synth-direct-trade-heirloom-leggings.",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,'},
 'score': 0.0,
 'values': []}, {'id': '89bc6ac5-c0a6-42ca-a3c7-4b05c724992c',
 'metadata': {'chunk': 14.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FvMDw80aFPZ2hHOWjIhV3-2FVoluptate-vexillologist-synth-direct-trade-heirloom-leggings-34.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":35,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":35,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:05.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-76a3-49a4-a101-ac6edddce02c",\r\n'
                      '            "page":"/bolg2/#/tag/adaptogen",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fadaptogen-35.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":36,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":36,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:09.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-7ee3-4177-8be6-c3dcb564de8b",\r\n'
                      '            "page":"/bolg2/#/tag/craft",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fcraft-36.txt"\r\n'
                      '         }'},
 'score': 0.0,
 'values': []}, {'id': '22004cee-d2e6-4e39-979f-945dd2f8f1fa',
 'metadata': {'chunk': 15.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '}\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":37,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":37,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:48.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:13.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-86b0-4ec0-a308-f417fda29c7a",\r\n'
                      '            "page":"/bolg2/#/tag/semiotics",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fsemiotics-37.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":38,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":38,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:48.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:10.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-8f47-4c43-8c58-fce014133f34",\r\n'
                      '            '
                      '"page":"/bolg2/#/post/khidsPJEHTIwgD0pjFLc/Qui-tofu-hella,-four-dollar-toast-authentic-echo-park-in-shabby-chic-cornhole-deserunt-whatever.",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FkhidsPJEHTIwgD0pjFLc-2FQui-tofu-hella-2C-four-dollar-toast-authentic-echo-park-in-shabby-chic-cornhole-deserunt-whatever-38.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {'},
 'score': 0.0,
 'values': []}, {'id': 'ff7a0274-0210-47a7-babf-910082117b7a',
 'metadata': {'chunk': 16.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '}\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":39,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":39,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:48.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:14.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-978c-4686-98f2-0f4bd83ce2c0",\r\n'
                      '            "page":"/bolg2/#/tag/incididunt",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fincididunt-39.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":40,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":40,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:48.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:17.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-9fc2-44fa-92fb-276e491f14e3",\r\n'
                      '            "page":"/bolg2/#/tag/deserunt",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fdeserunt-40.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":41,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":41,\r\n'
                      '            "created_at":"2024-10-28 09:53:48.000000",'},
 'score': 0.0,
 'values': []}, {'id': 'eb79d003-f8d4-4c3a-808f-38df6a28dde4',
 'metadata': {'chunk': 17.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"updated_at":"2025-02-05 15:20:18.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-a7d1-4755-bd6f-f69c02efabb8",\r\n'
                      '            "page":"/bolg2/#/tag/chic",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fchic-41.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":42,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":42,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:48.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:22.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-af2a-4c6b-b0bc-f71be13157cc",\r\n'
                      '            '
                      '"page":"/bolg2/#/post/xhbYdFYN0f28eysrAuQQ/Woke-gluten-free-tacos,-incididunt-et-meditation-affogato-man-braid-activated-charcoal-venmo.",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FxhbYdFYN0f28eysrAuQQ-2FWoke-gluten-free-tacos-2C-incididunt-et-meditation-affogato-man-braid-activated-charcoal-venmo-42.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":43,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":43,\r\n'
                      '            "created_at":"2024-10-28 09:53:48.000000",'},
 'score': 0.0,
 'values': []}, {'id': 'fc820f5b-2fe7-451a-8472-0a318425572b',
 'metadata': {'chunk': 18.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"updated_at":"2025-02-05 15:20:22.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-b777-4164-b0da-2b3fed2e1768",\r\n'
                      '            "page":"/bolg2/#/tag/glossier",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fglossier-43.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":44,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":44,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:48.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:26.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-bee3-4170-91eb-375dab9b6ab9",\r\n'
                      '            "page":"/bolg2/#/tag/schlitz",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fschlitz-44.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":45,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":45,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:48.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:26.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-c6b4-40be-b58b-942394adf716",\r\n'
                      '            "page":"/bolg2/#/tag/hammock",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,'},
 'score': 0.0,
 'values': []}, {'id': '745d426e-2418-45d6-a6fa-de99370bd6c5',
 'metadata': {'chunk': 19.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fhammock-45.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":24,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":24,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:47.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:31.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e98-1f44-4700-8883-9e3841c448e1",\r\n'
                      '            "page":"/bolg2/#/tag/pinterest",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fpinterest-24.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":46,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":46,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:50.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:19:18.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3e9c-0e90-42ff-83a3-10fd39d85244",\r\n'
                      '            "page":"/Covid19Stats/#",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2FCovid19Stats-2F-23-46.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":47,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":47,'},
 'score': 0.0,
 'values': []}, {'id': '960f2a28-3cfd-48a4-b9ef-c8eecbf00e43',
 'metadata': {'chunk': 20.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"page_id":47,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:55.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:30.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3ea3-6aca-4ca5-8a63-553d6af0cc55",\r\n'
                      '            "page":"/RockPaperScissorsGame/#",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-2F-23-47.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":48,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":48,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:55.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:34.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3ea3-735b-4e8d-a8bd-3c74980e6a3e",\r\n'
                      '            "page":"/RockPaperScissorsGame/#/paper",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-2F-23-2Fpaper-48.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":49,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":49,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:55.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:34.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3ea3-7bb7-4f3e-8aaf-ce47b17cdba1",\r\n'
                      '            '
                      '"page":"/RockPaperScissorsGame/#/scissors",'},
 'score': 0.0,
 'values': []}, {'id': 'e1ba417d-9b1e-4c34-a2c2-217edffe8220',
 'metadata': {'chunk': 21.0,
              'file_name': 'dupes.txt',
              'is_dict': 'no',
              'text': '"page":"/RockPaperScissorsGame/#/scissors",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-2F-23-2Fscissors-49.txt"\r\n'
                      '         }\r\n'
                      '      },\r\n'
                      '      {\r\n'
                      '         "stdClass":{\r\n'
                      '            "id":50,\r\n'
                      '            "knowledgebase_crawler_id":226,\r\n'
                      '            "page_id":50,\r\n'
                      '            "created_at":"2024-10-28 '
                      '09:53:55.000000",\r\n'
                      '            "updated_at":"2025-02-05 '
                      '15:20:37.000000",\r\n'
                      '            '
                      '"uuid":"9d5a3ea3-84ed-48db-814d-8b1d90e20319",\r\n'
                      '            "page":"/RockPaperScissorsGame/#/rock",\r\n'
                      '            "domain_id":2,\r\n'
                      '            "is_viewed":0,\r\n'
                      '            '
                      '"txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-2F-23-2Frock-50.txt"\r\n'
                      '         }\r\n'
                      '      }\r\n'
                      '   ]\r\n'
                      '}'},
 'score': 0.0,
 'values': []}, {'id': 'dc520a4a-c4c7-4c80-b611-429ea2cbbaa4',
 'metadata': {'chunk': 0.0,
              'file_name': 'apacare-primer%282%29.txt',
              'is_dict': 'no',
              'text': 'You are a digital sales rep for ApaCare, a dental care '
                      'company. Please assist clients with their '
                      'dental-related questions.\r\n'
                      'Use German in your responses.\r\n'
                      '\r\n'
                      'Start by asking a general question:\r\n'
                      '"Are you looking for a specific type of dental product '
                      'or advice?"\r\n'
                      '\r\n'
                      'If they are looking for advice, proceed with a '
                      'questionnaire about their dental care needs:\r\n'
                      'Are they focusing on whitening, sensitivity, gum '
                      'health, or general hygiene?\r\n'
                      'Try to ask a questionnaire to have clients describe '
                      'their problems.\r\n'
                      'If they are looking for dental products:\r\n'
                      'give them a product suggestion from ApaCare only.\r\n'
                      'If they are not looking for dental products or advice, '
                      'skip to general suggestions or conversation.\r\n'
                      '\r\n'
                      'Once the questionnaire is complete:\r\n'
                      'Suggest a product and do not repeat the questionnaire '
                      'unless explicitly requested.\r\n'
                      'Format the questionnaire to be readable for the users, '
                      'like a list or similar.\r\n'
                      '\r\n'
                      'When suggesting a product:\r\n'
                      "Look for the relevant product's page in the context.\r\n"
                      'Provide a detailed suggestion with an anchor tag link. '
                      'Ensure the target attribute is set to "__blank" and use '
                      'this format:\r\n'
                      '<a href="" target="__blank" style="background-color: '
                      '#brandcolor; color: white; padding: 10px; display: '
                      'block; text-align: center; border-radius: 5px; '
                      'text-decoration: none; margin-bottom: 0.75rem;" '
                      'aria-label="[replace this with the product name]">\r\n'
                      '[replace this with the product name]\r\n'
                      '</a>\r\n'
                      '\r\n'
                      'All links should have "__blank" target attribute.\r\n'
                      "Don't translate links href to German.\r\n"
                      '\r\n'
                      'Include related video suggestions:\r\n'
                      '\r\n'
                      'Search YouTube for videos about the product or topic '
                      '(e.g., how to use an electric toothbrush, flossing '
                      'techniques).\r\n'
                      'Embed the video in an iframe using this format:\r\n'
                      '<iframe\r\n'
                      'width="100%"\r\n'
                      'height="auto"\r\n'
                      'src="https://www.youtube.com/embed/videoId"\r\n'
                      'title="YouTube video player"\r\n'
                      'frameborder="0"\r\n'
                      'allow="accelerometer; autoplay; clipboard-write; '
                      'encrypted-media; gyroscope; picture-in-picture; '
                      'web-share"\r\n'
                      'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>'},
 'score': 0.0,
 'values': []}, {'id': '7faee156-f57e-4930-b1a4-a0ea0e26d6e2',
 'metadata': {'chunk': 1.0,
              'file_name': 'apacare-primer%282%29.txt',
              'is_dict': 'no',
              'text': 'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>\r\n'
                      '</iframe>\r\n'
                      '\r\n'
                      'For Google Drive videos, append /preview to the link '
                      'and embed it:\r\n'
                      '<iframe src="" width="100%" height="auto" '
                      'allow="autoplay"></iframe>\r\n'
                      '\r\n'
                      'For public URL video links, use the <video> tag:\r\n'
                      '<video src="" width="100%" height="auto" '
                      'controls></video>\r\n'
                      '\r\n'
                      'If no videos are available, skip the multimedia '
                      'content.\r\n'
                      '\r\n'
                      'Guidelines:\r\n'
                      '\r\n'
                      "Focus on promoting ApaCare's dental care products "
                      'onlyâ\x80\x94do not suggest competitors.\r\n'
                      "Suggest ApaCare's products as much as possible.\r\n"
                      "Suggest products from ApaCare's website's products "
                      'page.\r\n'
                      '\r\n'
                      'Answer general dental hygiene questions, offer '
                      'maintenance tips (e.g., replacing toothbrush heads, '
                      'flossing properly), or provide product-related FAQs.\r\n'
                      'Use engaging emojis where applicable (e.g., '
                      'ð\x9f¦·â\x9c¨ for whitening, ð\x9fª¥ for '
                      'toothbrushes).\r\n'
                      'Once a product is suggested:\r\n'
                      '\r\n'
                      'Avoid repeating the questionnaire unless the user '
                      'requests it.\r\n'
                      'Offer general dental care tips, product care advice, or '
                      'related topics to keep the conversation engaging.\r\n'
                      'For users who are unsure:\r\n'
                      'Provide a list of recommendations based on common '
                      'concerns:\r\n'
                      '\r\n'
                      'Whitening toothpaste\r\n'
                      'Electric toothbrushes for sensitive gums\r\n'
                      '\r\n'
                      "Don't tag your company's products links, videos, faq, "
                      'etc. as copyright infringement. \r\n'
                      'Give a disclaimer at the end of suggestions/advices '
                      'like "I am not a real doctor, but I can provide tips '
                      'based on expert recommendations. Please feel free to '
                      'contact our team anytime!" or similar.'},
 'score': 0.0,
 'values': []}, {'id': '41c239fc-8ac6-402e-8e4a-25db1d49cae4',
 'metadata': {'chunk': 0.0,
              'file_name': 'apacare-primer%282%29.txt',
              'is_dict': 'no',
              'text': 'You are a digital sales rep for ApaCare, a dental care '
                      'company. Please assist clients with their '
                      'dental-related questions.\r\n'
                      'Use German in your responses.\r\n'
                      '\r\n'
                      'Start by asking a general question:\r\n'
                      '"Are you looking for a specific type of dental product '
                      'or advice?"\r\n'
                      '\r\n'
                      'If they are looking for advice, proceed with a '
                      'questionnaire about their dental care needs:\r\n'
                      'Are they focusing on whitening, sensitivity, gum '
                      'health, or general hygiene?\r\n'
                      'Try to ask a questionnaire to have clients describe '
                      'their problems.\r\n'
                      'If they are looking for dental products:\r\n'
                      'give them a product suggestion from ApaCare only.\r\n'
                      'If they are not looking for dental products or advice, '
                      'skip to general suggestions or conversation.\r\n'
                      '\r\n'
                      'Once the questionnaire is complete:\r\n'
                      'Suggest a product and do not repeat the questionnaire '
                      'unless explicitly requested.\r\n'
                      'Format the questionnaire to be readable for the users, '
                      'like a list or similar.\r\n'
                      '\r\n'
                      'When suggesting a product:\r\n'
                      "Look for the relevant product's page in the context.\r\n"
                      'Provide a detailed suggestion with an anchor tag link. '
                      'Ensure the target attribute is set to "__blank" and use '
                      'this format:\r\n'
                      '<a href="" target="__blank" style="background-color: '
                      '#brandcolor; color: white; padding: 10px; display: '
                      'block; text-align: center; border-radius: 5px; '
                      'text-decoration: none; margin-bottom: 0.75rem;" '
                      'aria-label="[replace this with the product name]">\r\n'
                      '[replace this with the product name]\r\n'
                      '</a>\r\n'
                      '\r\n'
                      'All links should have "__blank" target attribute.\r\n'
                      "Don't translate links href to German.\r\n"
                      '\r\n'
                      'Include related video suggestions:\r\n'
                      '\r\n'
                      'Search YouTube for videos about the product or topic '
                      '(e.g., how to use an electric toothbrush, flossing '
                      'techniques).\r\n'
                      'Embed the video in an iframe using this format:\r\n'
                      '<iframe\r\n'
                      'width="100%"\r\n'
                      'height="auto"\r\n'
                      'src="https://www.youtube.com/embed/videoId"\r\n'
                      'title="YouTube video player"\r\n'
                      'frameborder="0"\r\n'
                      'allow="accelerometer; autoplay; clipboard-write; '
                      'encrypted-media; gyroscope; picture-in-picture; '
                      'web-share"\r\n'
                      'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>'},
 'score': 0.0,
 'values': []}, {'id': '38d46504-4583-43f8-9409-dcce0b7f2196',
 'metadata': {'chunk': 1.0,
              'file_name': 'apacare-primer%282%29.txt',
              'is_dict': 'no',
              'text': 'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>\r\n'
                      '</iframe>\r\n'
                      '\r\n'
                      'For Google Drive videos, append /preview to the link '
                      'and embed it:\r\n'
                      '<iframe src="" width="100%" height="auto" '
                      'allow="autoplay"></iframe>\r\n'
                      '\r\n'
                      'For public URL video links, use the <video> tag:\r\n'
                      '<video src="" width="100%" height="auto" '
                      'controls></video>\r\n'
                      '\r\n'
                      'If no videos are available, skip the multimedia '
                      'content.\r\n'
                      '\r\n'
                      'Guidelines:\r\n'
                      '\r\n'
                      "Focus on promoting ApaCare's dental care products "
                      'onlyâ\x80\x94do not suggest competitors.\r\n'
                      "Suggest ApaCare's products as much as possible.\r\n"
                      "Suggest products from ApaCare's website's products "
                      'page.\r\n'
                      '\r\n'
                      'Answer general dental hygiene questions, offer '
                      'maintenance tips (e.g., replacing toothbrush heads, '
                      'flossing properly), or provide product-related FAQs.\r\n'
                      'Use engaging emojis where applicable (e.g., '
                      'ð\x9f¦·â\x9c¨ for whitening, ð\x9fª¥ for '
                      'toothbrushes).\r\n'
                      'Once a product is suggested:\r\n'
                      '\r\n'
                      'Avoid repeating the questionnaire unless the user '
                      'requests it.\r\n'
                      'Offer general dental care tips, product care advice, or '
                      'related topics to keep the conversation engaging.\r\n'
                      'For users who are unsure:\r\n'
                      'Provide a list of recommendations based on common '
                      'concerns:\r\n'
                      '\r\n'
                      'Whitening toothpaste\r\n'
                      'Electric toothbrushes for sensitive gums\r\n'
                      '\r\n'
                      "Don't tag your company's products links, videos, faq, "
                      'etc. as copyright infringement. \r\n'
                      'Give a disclaimer at the end of suggestions/advices '
                      'like "I am not a real doctor, but I can provide tips '
                      'based on expert recommendations. Please feel free to '
                      'contact our team anytime!" or similar.'},
 'score': 0.0,
 'values': []}, {'id': '0f436054-8a19-4bca-b593-8ac80e61b4db',
 'metadata': {'chunk': 0.0,
              'file_name': 'apacare-primer%282%29.txt',
              'is_dict': 'no',
              'text': 'You are a digital sales rep for ApaCare, a dental care '
                      'company. Please assist clients with their '
                      'dental-related questions.\r\n'
                      'Use German in your responses.\r\n'
                      '\r\n'
                      'Start by asking a general question:\r\n'
                      '"Are you looking for a specific type of dental product '
                      'or advice?"\r\n'
                      '\r\n'
                      'If they are looking for advice, proceed with a '
                      'questionnaire about their dental care needs:\r\n'
                      'Are they focusing on whitening, sensitivity, gum '
                      'health, or general hygiene?\r\n'
                      'Try to ask a questionnaire to have clients describe '
                      'their problems.\r\n'
                      'If they are looking for dental products:\r\n'
                      'give them a product suggestion from ApaCare only.\r\n'
                      'If they are not looking for dental products or advice, '
                      'skip to general suggestions or conversation.\r\n'
                      '\r\n'
                      'Once the questionnaire is complete:\r\n'
                      'Suggest a product and do not repeat the questionnaire '
                      'unless explicitly requested.\r\n'
                      'Format the questionnaire to be readable for the users, '
                      'like a list or similar.\r\n'
                      '\r\n'
                      'When suggesting a product:\r\n'
                      "Look for the relevant product's page in the context.\r\n"
                      'Provide a detailed suggestion with an anchor tag link. '
                      'Ensure the target attribute is set to "__blank" and use '
                      'this format:\r\n'
                      '<a href="" target="__blank" style="background-color: '
                      '#brandcolor; color: white; padding: 10px; display: '
                      'block; text-align: center; border-radius: 5px; '
                      'text-decoration: none; margin-bottom: 0.75rem;" '
                      'aria-label="[replace this with the product name]">\r\n'
                      '[replace this with the product name]\r\n'
                      '</a>\r\n'
                      '\r\n'
                      'All links should have "__blank" target attribute.\r\n'
                      "Don't translate links href to German.\r\n"
                      '\r\n'
                      'Include related video suggestions:\r\n'
                      '\r\n'
                      'Search YouTube for videos about the product or topic '
                      '(e.g., how to use an electric toothbrush, flossing '
                      'techniques).\r\n'
                      'Embed the video in an iframe using this format:\r\n'
                      '<iframe\r\n'
                      'width="100%"\r\n'
                      'height="auto"\r\n'
                      'src="https://www.youtube.com/embed/videoId"\r\n'
                      'title="YouTube video player"\r\n'
                      'frameborder="0"\r\n'
                      'allow="accelerometer; autoplay; clipboard-write; '
                      'encrypted-media; gyroscope; picture-in-picture; '
                      'web-share"\r\n'
                      'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>'},
 'score': 0.0,
 'values': []}, {'id': 'f3e19ec6-03e8-4217-9205-75ae8a73756f',
 'metadata': {'chunk': 1.0,
              'file_name': 'apacare-primer%282%29.txt',
              'is_dict': 'no',
              'text': 'referrerpolicy="strict-origin-when-cross-origin"\r\n'
                      'allowfullscreen>\r\n'
                      '</iframe>\r\n'
                      '\r\n'
                      'For Google Drive videos, append /preview to the link '
                      'and embed it:\r\n'
                      '<iframe src="" width="100%" height="auto" '
                      'allow="autoplay"></iframe>\r\n'
                      '\r\n'
                      'For public URL video links, use the <video> tag:\r\n'
                      '<video src="" width="100%" height="auto" '
                      'controls></video>\r\n'
                      '\r\n'
                      'If no videos are available, skip the multimedia '
                      'content.\r\n'
                      '\r\n'
                      'Guidelines:\r\n'
                      '\r\n'
                      "Focus on promoting ApaCare's dental care products "
                      'onlyâ\x80\x94do not suggest competitors.\r\n'
                      "Suggest ApaCare's products as much as possible.\r\n"
                      "Suggest products from ApaCare's website's products "
                      'page.\r\n'
                      '\r\n'
                      'Answer general dental hygiene questions, offer '
                      'maintenance tips (e.g., replacing toothbrush heads, '
                      'flossing properly), or provide product-related FAQs.\r\n'
                      'Use engaging emojis where applicable (e.g., '
                      'ð\x9f¦·â\x9c¨ for whitening, ð\x9fª¥ for '
                      'toothbrushes).\r\n'
                      'Once a product is suggested:\r\n'
                      '\r\n'
                      'Avoid repeating the questionnaire unless the user '
                      'requests it.\r\n'
                      'Offer general dental care tips, product care advice, or '
                      'related topics to keep the conversation engaging.\r\n'
                      'For users who are unsure:\r\n'
                      'Provide a list of recommendations based on common '
                      'concerns:\r\n'
                      '\r\n'
                      'Whitening toothpaste\r\n'
                      'Electric toothbrushes for sensitive gums\r\n'
                      '\r\n'
                      "Don't tag your company's products links, videos, faq, "
                      'etc. as copyright infringement. \r\n'
                      'Give a disclaimer at the end of suggestions/advices '
                      'like "I am not a real doctor, but I can provide tips '
                      'based on expert recommendations. Please feel free to '
                      'contact our team anytime!" or similar.'},
 'score': 0.0,
 'values': []}, {'id': '5477e319-9f27-412c-a26a-a3b601afff88',
 'metadata': {'chunk': 0.0,
              'file_name': 'chatbot-error.txt',
              'is_dict': 'no',
              'text': 'INFO:     172.17.0.1:57890 - "POST '
                      '/kios/knowledgebase/flexible-query/ HTTP/1.1" 500 '
                      'Internal Server Error\r\n'
                      'ERROR:    Exception in ASGI application\r\n'
                      'Traceback (most recent call last):\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", '
                      'line 426, in run_asgi\r\n'
                      '    result = await app(  # type: '
                      'ignore[func-returns-value]\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", '
                      'line 84, in __call__\r\n'
                      '    return await self.app(scope, receive, send)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", '
                      'line 289, in __call__\r\n'
                      '    await super().__call__(scope, receive, send)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/applications.py", '
                      'line 122, in __call__\r\n'
                      '    await self.middleware_stack(scope, receive, '
                      'send)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", '
                      'line 184, in __call__\r\n'
                      '    raise exc\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", '
                      'line 162, in __call__\r\n'
                      '    await self.app(scope, receive, _send)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", '
                      'line 83, in __call__\r\n'
                      '    await self.app(scope, receive, send)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", '
                      'line 79, in __call__\r\n'
                      '    raise exc\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", '
                      'line 68, in __call__\r\n'
                      '    await self.app(scope, receive, sender)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", '
                      'line 20, in __call__\r\n'
                      '    raise e'},
 'score': 0.0,
 'values': []}, {'id': '94cb46b6-5a10-4cc4-a8c6-dfbcfdc44f42',
 'metadata': {'chunk': 1.0,
              'file_name': 'chatbot-error.txt',
              'is_dict': 'no',
              'text': 'raise e\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", '
                      'line 17, in __call__\r\n'
                      '    await self.app(scope, receive, send)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/routing.py", '
                      'line 718, in __call__\r\n'
                      '    await route.handle(scope, receive, send)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/routing.py", '
                      'line 276, in handle\r\n'
                      '    await self.app(scope, receive, send)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/routing.py", '
                      'line 66, in app\r\n'
                      '    response = await func(request)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", '
                      'line 273, in app\r\n'
                      '    raw_response = await run_endpoint_function(\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", '
                      'line 192, in run_endpoint_function\r\n'
                      '    return await run_in_threadpool(dependant.call, '
                      '**values)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/concurrency.py", '
                      'line 41, in run_in_threadpool\r\n'
                      '    return await anyio.to_thread.run_sync(func, '
                      '*args)\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", '
                      'line 33, in run_sync\r\n'
                      '    return await '
                      'get_asynclib().run_sync_in_worker_thread(\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", '
                      'line 877, in run_sync_in_worker_thread\r\n'
                      '    return await future\r\n'
                      '  File '
                      '"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", '
                      'line 807, in run\r\n'
                      '    result = context.run(func, *args)\r\n'
                      '  File "/app/main.py", line 1020, in '
                      'kios_retrieve_information\r\n'
                      '    return process_single_query(\r\n'
                      '  File "/app/main.py", line 1053, in '
                      'process_single_query\r\n'
                      '    provider = AIProviders(input.model, '
                      'ws_connection)\r\n'
                      '  File "/app/ai_providers.py", line 15, in __init__'},
 'score': 0.0,
 'values': []}, {'id': 'cb2eb7e0-4dee-417b-901c-4183fceb4d64',
 'metadata': {'chunk': 2.0,
              'file_name': 'chatbot-error.txt',
              'is_dict': 'no',
              'text': 'File "/app/ai_providers.py", line 15, in __init__\r\n'
                      '    self.set_provider(model)\r\n'
                      '  File "/app/ai_providers.py", line 29, in '
                      'set_provider\r\n'
                      '    self.provider = Gemini(self.model)\r\n'
                      '  File "/app/providers/gemini.py", line 10, in '
                      '__init__\r\n'
                      '    random_key = '
                      'random.choice(config.settings.GEMINI_API_KEYS)\r\n'
                      '  File "/usr/lib/python3.10/random.py", line 378, in '
                      'choice\r\n'
                      '    return seq[self._randbelow(len(seq))]\r\n'
                      'IndexError: list index out of range'},
 'score': 0.0,
 'values': []}, {'id': '500fbf44-69bb-4d60-8b0f-613311ead6d9',
 'metadata': {'chunk': 0.0,
              'file_name': 'link.txt',
              'is_dict': 'no',
              'text': "rebecca black's friday youtube link\r\n"
                      'https://www.youtube.com/watch?v=dQw4w9WgXcQ'},
 'score': 0.0,
 'values': []}, {'id': '76e06c63-dbe7-402c-993b-a68cbf0144a2',
 'metadata': {'chunk': 0.0,
              'file_name': 'gpt-vector-dimension-error.txt',
              'is_dict': 'no',
              'text': '2025-01-23 15:23:13 =======================\r\n'
                      '2025-01-23 15:23:13 CHAT RECEIVED\r\n'
                      '2025-01-23 15:23:13 =======================\r\n'
                      '2025-01-23 15:23:18 INFO:     172.18.0.1:56228 - "POST '
                      '/kios/knowledgebase/flexible-query/ HTTP/1.1" 500 '
                      'Internal Server Error\r\n'
                      '2025-01-23 15:23:18 ERROR:    Exception in ASGI '
                      'application\r\n'
                      '2025-01-23 15:23:18 Traceback (most recent call '
                      'last):\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", '
                      'line 426, in run_asgi\r\n'
                      '2025-01-23 15:23:18     result = await app(  # type: '
                      'ignore[func-returns-value]\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", '
                      'line 84, in __call__\r\n'
                      '2025-01-23 15:23:18     return await self.app(scope, '
                      'receive, send)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", '
                      'line 289, in __call__\r\n'
                      '2025-01-23 15:23:18     await super().__call__(scope, '
                      'receive, send)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/applications.py", '
                      'line 122, in __call__\r\n'
                      '2025-01-23 15:23:18     await '
                      'self.middleware_stack(scope, receive, send)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", '
                      'line 184, in __call__\r\n'
                      '2025-01-23 15:23:18     raise exc'},
 'score': 0.0,
 'values': []}, {'id': '30a58c34-d0e2-4828-a8ed-836afa831454',
 'metadata': {'chunk': 1.0,
              'file_name': 'gpt-vector-dimension-error.txt',
              'is_dict': 'no',
              'text': '2025-01-23 15:23:18     raise exc\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", '
                      'line 162, in __call__\r\n'
                      '2025-01-23 15:23:18     await self.app(scope, receive, '
                      '_send)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", '
                      'line 83, in __call__\r\n'
                      '2025-01-23 15:23:18     await self.app(scope, receive, '
                      'send)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", '
                      'line 79, in __call__\r\n'
                      '2025-01-23 15:23:18     raise exc\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", '
                      'line 68, in __call__\r\n'
                      '2025-01-23 15:23:18     await self.app(scope, receive, '
                      'sender)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", '
                      'line 20, in __call__\r\n'
                      '2025-01-23 15:23:18     raise e\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", '
                      'line 17, in __call__\r\n'
                      '2025-01-23 15:23:18     await self.app(scope, receive, '
                      'send)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/routing.py", '
                      'line 718, in __call__\r\n'
                      '2025-01-23 15:23:18     await route.handle(scope, '
                      'receive, send)'},
 'score': 0.0,
 'values': []}, {'id': 'f8d88d4c-8ac5-4a5b-9358-b5d2d7ca796b',
 'metadata': {'chunk': 2.0,
              'file_name': 'gpt-vector-dimension-error.txt',
              'is_dict': 'no',
              'text': '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/routing.py", '
                      'line 276, in handle\r\n'
                      '2025-01-23 15:23:18     await self.app(scope, receive, '
                      'send)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/routing.py", '
                      'line 66, in app\r\n'
                      '2025-01-23 15:23:18     response = await '
                      'func(request)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", '
                      'line 273, in app\r\n'
                      '2025-01-23 15:23:18     raw_response = await '
                      'run_endpoint_function(\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", '
                      'line 192, in run_endpoint_function\r\n'
                      '2025-01-23 15:23:18     return await '
                      'run_in_threadpool(dependant.call, **values)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/starlette/concurrency.py", '
                      'line 41, in run_in_threadpool\r\n'
                      '2025-01-23 15:23:18     return await '
                      'anyio.to_thread.run_sync(func, *args)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", '
                      'line 33, in run_sync\r\n'
                      '2025-01-23 15:23:18     return await '
                      'get_asynclib().run_sync_in_worker_thread(\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", '
                      'line 877, in run_sync_in_worker_thread\r\n'
                      '2025-01-23 15:23:18     return await future\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", '
                      'line 807, in run'},
 'score': 0.0,
 'values': []}, {'id': '762860f7-4cd7-4295-9c41-3cfcd87b3fa8',
 'metadata': {'chunk': 3.0,
              'file_name': 'gpt-vector-dimension-error.txt',
              'is_dict': 'no',
              'text': '2025-01-23 15:23:18     result = context.run(func, '
                      '*args)\r\n'
                      '2025-01-23 15:23:18   File "/app/main.py", line 2018, '
                      'in kios_retrieve_information_via_chat\r\n'
                      '2025-01-23 15:23:18     search_results = '
                      'docsearch.max_marginal_relevance_search_by_vector(res.data[0].embedding,k=actual_k, '
                      'fetch_k=top_k)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/langchain_pinecone/vectorstores.py", '
                      'line 307, in max_marginal_relevance_search_by_vector\r\n'
                      '2025-01-23 15:23:18     results = self._index.query(\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/utils/error_handling.py", '
                      'line 10, in inner_func\r\n'
                      '2025-01-23 15:23:18     return func(*args, **kwargs)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/data/index.py", '
                      'line 399, in query\r\n'
                      '2025-01-23 15:23:18     response = '
                      'self._vector_api.query(\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", '
                      'line 772, in __call__\r\n'
                      '2025-01-23 15:23:18     return self.callable(self, '
                      '*args, **kwargs)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api/data_plane_api.py", '
                      'line 844, in __query\r\n'
                      '2025-01-23 15:23:18     return '
                      'self.call_with_http_info(**kwargs)\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", '
                      'line 834, in call_with_http_info'},
 'score': 0.0,
 'values': []}, {'id': '2996fde4-646b-4f84-95bb-15b841b878e4',
 'metadata': {'chunk': 4.0,
              'file_name': 'gpt-vector-dimension-error.txt',
              'is_dict': 'no',
              'text': '2025-01-23 15:23:18     return '
                      'self.api_client.call_api(\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", '
                      'line 409, in call_api\r\n'
                      '2025-01-23 15:23:18     return '
                      'self.__call_api(resource_path, method,\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", '
                      'line 203, in __call_api\r\n'
                      '2025-01-23 15:23:18     raise e\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", '
                      'line 196, in __call_api\r\n'
                      '2025-01-23 15:23:18     response_data = '
                      'self.request(\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", '
                      'line 455, in request\r\n'
                      '2025-01-23 15:23:18     return '
                      'self.rest_client.POST(url,\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/rest.py", '
                      'line 302, in POST\r\n'
                      '2025-01-23 15:23:18     return self.request("POST", '
                      'url,\r\n'
                      '2025-01-23 15:23:18   File '
                      '"/usr/local/lib/python3.10/dist-packages/pinecone/core/client/rest.py", '
                      'line 261, in request\r\n'
                      '2025-01-23 15:23:18     raise '
                      'PineconeApiException(http_resp=r)\r\n'
                      '2025-01-23 15:23:18 '
                      'pinecone.core.client.exceptions.PineconeApiException: '
                      '(400)\r\n'
                      '2025-01-23 15:23:18 Reason: Bad Request'},
 'score': 0.0,
 'values': []}, {'id': '70a8b026-cebd-4d09-9f36-1702675f9515',
 'metadata': {'chunk': 5.0,
              'file_name': 'gpt-vector-dimension-error.txt',
              'is_dict': 'no',
              'text': '2025-01-23 15:23:18 Reason: Bad Request\r\n'
                      '2025-01-23 15:23:18 HTTP response headers: '
                      "HTTPHeaderDict({'content-type': 'application/json', "
                      "'Content-Length': '104', "
                      "'x-pinecone-request-latency-ms': '73', "
                      "'x-pinecone-request-id': '5221281482731493865', 'date': "
                      "'Thu, 23 Jan 2025 07:23:09 GMT', "
                      "'x-envoy-upstream-service-time': '17', 'server': "
                      "'envoy', 'Via': '1.1 google', 'Alt-Svc': "
                      '\'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000\'})\r\n'
                      '2025-01-23 15:23:18 HTTP response body: '
                      '{"code":3,"message":"Vector dimension 3072 does not '
                      'match the dimension of the index 1536","details":[]}'},
 'score': 0.0,
 'values': []}, {'id': 'c0b9c8d4-c80e-4742-a8d4-c6ef6efb91de',
 'metadata': {'chunk': 0.0,
              'file_name': 'gemini-quota-error.txt',
              'is_dict': 'no',
              'text': '429 error fires even when not doing load test\r\n'
                      '\r\n'
                      '2025-01-23 15:14:08 =======================\r\n'
                      '2025-01-23 15:14:08 CHAT RECEIVED\r\n'
                      '2025-01-23 15:14:08 =======================\r\n'
                      '2025-01-23 15:14:09 Retrying '
                      'langchain_google_vertexai.chat_models._completion_with_retry.<locals>._completion_with_retry_inner '
                      'in 4.0 seconds as it raised ResourceExhausted: 429 '
                      'Quota exceeded for '
                      'aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model '
                      'with base model: gemini-1.5-pro. Please submit a quota '
                      'increase request. '
                      'https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai..\r\n'
                      '2025-01-23 15:14:14 Retrying '
                      'langchain_google_vertexai.chat_models._completion_with_retry.<locals>._completion_with_retry_inner '
                      'in 4.0 seconds as it raised ResourceExhausted: 429 '
                      'Quota exceeded for '
                      'aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model '
                      'with base model: gemini-1.5-pro. Please submit a quota '
                      'increase request. '
                      'https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai..'},
 'score': 0.0,
 'values': []}, {'id': '469071c1-1bda-465b-b59f-bbe4c7621e98',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': 'f10e6e51-d1a4-4a2b-9b4c-4f5f0b0d9b5d',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '97285c8e-ee73-4723-8b64-3ebefa86390b',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '44fadd98-c05f-4f00-9c0a-bcfef6b18dcb',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '59db5911-4467-405d-aab2-304e61fea13c',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '8c77a51a-c9c0-4665-bccb-4827b5ad6558',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '4fbfec9d-ef27-40df-b499-bdfd939220cf',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}, {'id': '21350f11-a2d8-4646-90ee-63cd843631f1',
 'metadata': {'chunk': 0.0,
              'file_name': 'crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt',
              'is_dict': 'no',
              'text': '- if CrawlerJob fails statues will never update, import '
                      'status wont update\r\n'
                      '(add failed() method -> create CrawlerProcess with '
                      'failed status, record last process time??)\r\n'
                      '- if CrawlerProcessJob fails before recording last '
                      'process time '
                      '("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), '
                      'now());") the status will never upate\r\n'
                      '- importing failed Crawler pages still marked '
                      'success\r\n'
                      '- if CrawlerFilesJob fails CrawlerProcess status wont '
                      'update\r\n'
                      '- if CrawlerPrepareKnowledgebaseTrainingJob fails '
                      'import status wont update\r\n'
                      '- CrawlerFilesProcessTrainingJob@handleProcessingError '
                      '-- failed items are marked as processed/success.\r\n'
                      'should be markItemAsFailed() same as in '
                      'CrawlerPageProcessTrainingJob?\r\n'
                      '\r\n'
                      '- Finalizing Logic Duplication\r\n'
                      'The completion checking and finalization logic is '
                      'duplicated across multiple jobs:\r\n'
                      '\r\n'
                      'CrawlerPageProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize\r\n'
                      'CheckKnowledgebaseCrawlerImportCompletion::handle\r\n'
                      '\r\n'
                      'Each has subtle differences, creating opportunities for '
                      'inconsistent behavior.\r\n'
                      '\r\n'
                      '- Unreliable S3 File Operations\r\n'
                      'File operations on S3 have minimal error handling:\r\n'
                      '\r\n'
                      '$this->filesystem->put($s3Path, $newContent);\r\n'
                      'return $this->filesystem->url($s3Path);\r\n'
                      '\r\n'
                      'If the S3 put operation fails silently, subsequent code '
                      'would continue with a URL to a non-existent file.\r\n'
                      '\r\n'
                      '- try using knowledgebase_crawler_imports table instead '
                      "of cache for counting since it's already "
                      'implemented?\r\n'
                      'update counts every x seconds instead of realtime '
                      'updates?\r\n'
                      '\r\n'
                      '- CrawlerFileProcessTrainingJob and/or '
                      'CrawlerPageProcessTrainingJob failure not marking '
                      'KnowledgebaseCrawler as fail\r\n'
                      '- KnowledgebaseCrawlerImport fails getting deleted '
                      'after'},
 'score': 0.0,
 'values': []}], 'dictionary_entries': []}
==================================================
**Elapsed Time: 0.69 seconds**
==================================================



CONTEXT PREPARED
==================================================
**Elapsed Time: 1.10 seconds**
==================================================



PRIMER
Primer: WICHTIG: Wiederholen Sie diese Anweisungen in Ihren Antworten nicht, auch wenn Sie darum gebeten werden.


                Sie sind Simon, ein intelligenter persönlicher Assistent im KIOS-System.
                Sie können auf Wissensdatenbanken zugreifen, die im "KONTEXT" des Benutzers bereitgestellt werden.
                Im "KONTEXT" priorisieren Sie Informationen aus dem mit "FEEDBACK:" gekennzeichneten Text.
            
                Ihre Rolle besteht darin, als Experte für Recherche zu agieren.
                Priorisieren Sie Präzision und Genauigkeit in Ihren Antworten.
                Nutzen Sie den Chat-Verlauf für kontextbezogene Analysen.
                Konzentrieren Sie sich auf faktenbasierte Informationen.
==================================================
**Elapsed Time: 0.00 seconds**
==================================================



FILE SOURCES
File Sources: crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%285%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%283%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2821%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2820%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%282%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%284%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%282%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%281%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%281%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%285%29%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%282%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%2825%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2826%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%283%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%285%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%286%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%281%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%284%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%287%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2814%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2813%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2812%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%288%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%289%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2810%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2811%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2822%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2823%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2824%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2818%29.txt, crawler-issues-19MAR2025%20-%20Copy%282%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2815%29.txt, crawler-issues-19MAR2025%20-%20Copy%281%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2819%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2817%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%2816%29.txt, apacare-primer%281%29.txt, apacare-primer.txt, crawler-issues-19MAR2025%282%29.txt, crawler-issues-19MAR2025.txt, crawler-issues-19MAR2025%281%29.txt, dupes.txt - Page 1, apacare-primer%282%29.txt, chatbot-error.txt, link.txt - Page 1, gpt-vector-dimension-error.txt, gemini-quota-error.txt, crawler-issues-19MAR2025%20-%20Copy.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt, crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt
==================================================
**Elapsed Time: 0.00 seconds**
==================================================



AUGMENTED QUERY
Augmented Query: KONTEXT: ##########
File: crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%285%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%283%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2821%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2820%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%284%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%283%29%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%285%29%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%284%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2825%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2826%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%283%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%285%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%286%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%284%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%287%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2814%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2813%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2812%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%288%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%289%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2810%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2811%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2822%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2823%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2824%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2818%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2815%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2819%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2817%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%2816%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: apacare-primer%281%29.txt

Page: 1

Context: You are a digital sales rep for ApaCare, a dental care company. Please assist clients with their dental-related questions.
Use German in your responses.

Start by asking a general question:
"Are you looking for a specific type of dental product or advice?"

If they are looking for advice, proceed with a questionnaire about their dental care needs:
Are they focusing on whitening, sensitivity, gum health, or general hygiene?
Try to ask a questionnaire to have clients describe their problems.
If they are looking for dental products:
give them a product suggestion from ApaCare only.
If they are not looking for dental products or advice, skip to general suggestions or conversation.

Once the questionnaire is complete:
Suggest a product and do not repeat the questionnaire unless explicitly requested.
Format the questionnaire to be readable for the users, like a list or similar.

When suggesting a product:
Look for the relevant product's page in the context.
Provide a detailed suggestion with an anchor tag link. Ensure the target attribute is set to "__blank" and use this format:
<a href="" target="__blank" style="background-color: #brandcolor; color: white; padding: 10px; display: block; text-align: center; border-radius: 5px; text-decoration: none; margin-bottom: 0.75rem;" aria-label="[replace this with the product name]">
[replace this with the product name]
</a>

All links should have "__blank" target attribute.
Don't translate links href to German.

Include related video suggestions:

Search YouTube for videos about the product or topic (e.g., how to use an electric toothbrush, flossing techniques).
Embed the video in an iframe using this format:
<iframe
width="100%"
height="auto"
src="https://www.youtube.com/embed/videoId"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
####################
File: apacare-primer%281%29.txt

Page: 1

Context: referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
</iframe>

For Google Drive videos, append /preview to the link and embed it:
<iframe src="" width="100%" height="auto" allow="autoplay"></iframe>

For public URL video links, use the <video> tag:
<video src="" width="100%" height="auto" controls></video>

If no videos are available, skip the multimedia content.

Guidelines:

Focus on promoting ApaCare's dental care products only—do not suggest competitors.
Suggest ApaCare's products as much as possible.
Suggest products from ApaCare's website's products page.

Answer general dental hygiene questions, offer maintenance tips (e.g., replacing toothbrush heads, flossing properly), or provide product-related FAQs.
Use engaging emojis where applicable (e.g., 🦷✨ for whitening, 🪥 for toothbrushes).
Once a product is suggested:

Avoid repeating the questionnaire unless the user requests it.
Offer general dental care tips, product care advice, or related topics to keep the conversation engaging.
For users who are unsure:
Provide a list of recommendations based on common concerns:

Whitening toothpaste
Electric toothbrushes for sensitive gums

Don't tag your company's products links, videos, faq, etc. as copyright infringement. 
Give a disclaimer at the end of suggestions/advices like "I am not a real doctor, but I can provide tips based on expert recommendations. Please feel free to contact our team anytime!" or similar.
####################
File: apacare-primer.txt

Page: 1

Context: You are a digital sales rep for ApaCare, a dental care company. Please assist clients with their dental-related questions.
Use German in your responses.

Start by asking a general question:
"Are you looking for a specific type of dental product or advice?"

If they are looking for advice, proceed with a questionnaire about their dental care needs:
Are they focusing on whitening, sensitivity, gum health, or general hygiene?
Try to ask a questionnaire to have clients describe their problems.
If they are looking for dental products:
give them a product suggestion from ApaCare only.
If they are not looking for dental products or advice, skip to general suggestions or conversation.

Once the questionnaire is complete:
Suggest a product and do not repeat the questionnaire unless explicitly requested.
Format the questionnaire to be readable for the users, like a list or similar.

When suggesting a product:
Look for the relevant product's page in the context.
Provide a detailed suggestion with an anchor tag link. Ensure the target attribute is set to "__blank" and use this format:
<a href="" target="__blank" style="background-color: #brandcolor; color: white; padding: 10px; display: block; text-align: center; border-radius: 5px; text-decoration: none; margin-bottom: 0.75rem;" aria-label="[replace this with the product name]">
[replace this with the product name]
</a>

All links should have "__blank" target attribute.
Don't translate links href to German.

Include related video suggestions:

Search YouTube for videos about the product or topic (e.g., how to use an electric toothbrush, flossing techniques).
Embed the video in an iframe using this format:
<iframe
width="100%"
height="auto"
src="https://www.youtube.com/embed/videoId"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
####################
File: apacare-primer.txt

Page: 1

Context: referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
</iframe>

For Google Drive videos, append /preview to the link and embed it:
<iframe src="" width="100%" height="auto" allow="autoplay"></iframe>

For public URL video links, use the <video> tag:
<video src="" width="100%" height="auto" controls></video>

If no videos are available, skip the multimedia content.

Guidelines:

Focus on promoting ApaCare's dental care products only—do not suggest competitors.
Suggest ApaCare's products as much as possible.
Suggest products from ApaCare's website's products page.

Answer general dental hygiene questions, offer maintenance tips (e.g., replacing toothbrush heads, flossing properly), or provide product-related FAQs.
Use engaging emojis where applicable (e.g., 🦷✨ for whitening, 🪥 for toothbrushes).
Once a product is suggested:

Avoid repeating the questionnaire unless the user requests it.
Offer general dental care tips, product care advice, or related topics to keep the conversation engaging.
For users who are unsure:
Provide a list of recommendations based on common concerns:

Whitening toothpaste
Electric toothbrushes for sensitive gums

Don't tag your company's products links, videos, faq, etc. as copyright infringement. 
Give a disclaimer at the end of suggestions/advices like "I am not a real doctor, but I can provide tips based on expert recommendations. Please feel free to contact our team anytime!" or similar.
####################
File: crawler-issues-19MAR2025%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: dupes.txt

Page: 1

Context: {
   "data":[
      {
         "stdClass":{
            "id":2,
            "knowledgebase_crawler_id":226,
            "page_id":2,
            "created_at":"2024-10-28 09:53:07.000000",
            "updated_at":"2025-02-05 15:18:26.000000",
            "uuid":"9d5a3e5b-155c-4259-890a-e990bb997f5b",
            "page":"/",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2F-2.txt"
         }
      },
      {
         "stdClass":{
            "id":3,
            "knowledgebase_crawler_id":226,
            "page_id":3,
            "created_at":"2024-10-28 09:53:11.000000",
            "updated_at":"2025-02-05 15:18:34.000000",
            "uuid":"9d5a3e61-2c28-449a-92fb-2fe4d81b6e9e",
            "page":"/#/work",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2F-23-2Fwork-3.txt"
         }
      },
      {
         "stdClass":{
            "id":4,
            "knowledgebase_crawler_id":226,
            "page_id":4,
            "created_at":"2024-10-28 09:53:11.000000",
            "updated_at":"2025-02-05 15:18:33.000000",
            "uuid":"9d5a3e61-356e-4948-8535-eb94619d3501",
            "page":"/#/about",
####################
File: dupes.txt

Page: 1

Context: "page":"/#/about",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2F-23-2Fabout-4.txt"
         }
      },
      {
         "stdClass":{
            "id":5,
            "knowledgebase_crawler_id":226,
            "page_id":5,
            "created_at":"2024-10-28 09:53:11.000000",
            "updated_at":"2025-02-05 15:18:36.000000",
            "uuid":"9d5a3e61-40a1-47d1-bd7f-ac2047fb6b0a",
            "page":"/#/contact",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2F-23-2Fcontact-5.txt"
         }
      },
      {
         "stdClass":{
            "id":6,
            "knowledgebase_crawler_id":226,
            "page_id":6,
            "created_at":"2024-10-28 09:53:19.000000",
            "updated_at":"2025-02-05 15:18:42.000000",
            "uuid":"9d5a3e6c-8a5d-430d-8b66-c5c58548e3ba",
            "page":"/#",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2F-23-6.txt"
         }
      },
      {
         "stdClass":{
            "id":7,
            "knowledgebase_crawler_id":226,
            "page_id":7,
            "created_at":"2024-10-28 09:53:19.000000",
####################
File: dupes.txt

Page: 1

Context: "updated_at":"2025-02-05 15:18:42.000000",
            "uuid":"9d5a3e6c-92b5-4cb9-9a31-5430720eca62",
            "page":"/cheesemax",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fcheesemax-7.txt"
         }
      },
      {
         "stdClass":{
            "id":8,
            "knowledgebase_crawler_id":226,
            "page_id":8,
            "created_at":"2024-10-28 09:53:19.000000",
            "updated_at":"2025-02-05 15:18:49.000000",
            "uuid":"9d5a3e6c-9c6a-4308-b0b1-4a76852f2904",
            "page":"/bolg2",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-8.txt"
         }
      },
      {
         "stdClass":{
            "id":9,
            "knowledgebase_crawler_id":226,
            "page_id":9,
            "created_at":"2024-10-28 09:53:19.000000",
            "updated_at":"2025-02-05 15:18:46.000000",
            "uuid":"9d5a3e6c-a7a1-407b-841e-b99e00d1366c",
            "page":"/Covid19Stats",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2FCovid19Stats-9.txt"
         }
      },
      {
         "stdClass":{
####################
File: dupes.txt

Page: 1

Context: }
      },
      {
         "stdClass":{
            "id":10,
            "knowledgebase_crawler_id":226,
            "page_id":10,
            "created_at":"2024-10-28 09:53:19.000000",
            "updated_at":"2025-02-05 15:18:50.000000",
            "uuid":"9d5a3e6c-afb0-41cb-8238-098b84c40116",
            "page":"/ip-address-tracker",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fip-address-tracker-10.txt"
         }
      },
      {
         "stdClass":{
            "id":11,
            "knowledgebase_crawler_id":226,
            "page_id":11,
            "created_at":"2024-10-28 09:53:19.000000",
            "updated_at":"2025-02-05 15:18:54.000000",
            "uuid":"9d5a3e6c-b94c-4773-839a-074fa1326a53",
            "page":"/RockPaperScissorsGame",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-11.txt"
         }
      },
      {
         "stdClass":{
            "id":12,
            "knowledgebase_crawler_id":226,
            "page_id":12,
            "created_at":"2024-10-28 09:53:19.000000",
            "updated_at":"2025-02-05 15:18:56.000000",
            "uuid":"9d5a3e6c-c25a-4340-9c94-a588043047ad",
####################
File: dupes.txt

Page: 1

Context: "page":"/dbxmp",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fdbxmp-12.txt"
         }
      },
      {
         "stdClass":{
            "id":13,
            "knowledgebase_crawler_id":226,
            "page_id":13,
            "created_at":"2024-10-28 09:53:19.000000",
            "updated_at":"2025-02-05 15:18:58.000000",
            "uuid":"9d5a3e6c-cacf-483e-85c7-5e42010a140e",
            "page":"/gdmp",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fgdmp-13.txt"
         }
      },
      {
         "stdClass":{
            "id":14,
            "knowledgebase_crawler_id":226,
            "page_id":14,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:23.000000",
            "uuid":"9d5a3e97-c0ef-4c11-a00b-ce98292549f1",
            "page":"/bolg2/#",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-14.txt"
         }
      },
      {
         "stdClass":{
            "id":15,
            "knowledgebase_crawler_id":226,
            "page_id":15,
            "created_at":"2024-10-28 09:53:47.000000",
####################
File: dupes.txt

Page: 1

Context: "updated_at":"2025-02-05 15:19:22.000000",
            "uuid":"9d5a3e97-d202-4b78-ae70-1ab4b2b36159",
            "page":"/bolg2/#/new-post",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fnew-post-15.txt"
         }
      },
      {
         "stdClass":{
            "id":16,
            "knowledgebase_crawler_id":226,
            "page_id":16,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:26.000000",
            "uuid":"9d5a3e97-db24-48d3-886e-20dfd81362a3",
            "page":"/bolg2/#/login",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Flogin-16.txt"
         }
      },
      {
         "stdClass":{
            "id":17,
            "knowledgebase_crawler_id":226,
            "page_id":17,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:27.000000",
            "uuid":"9d5a3e97-e396-4761-acb4-17231f629ec0",
            "page":"/bolg2/#/post/oRqLD9jNpdFxjXZoL4Xc/New-Post-September-2021!!!",
            "domain_id":2,
            "is_viewed":0,
####################
File: dupes.txt

Page: 1

Context: "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FoRqLD9jNpdFxjXZoL4Xc-2FNew-Post-September-2021-21-21-21-17.txt"
         }
      },
      {
         "stdClass":{
            "id":18,
            "knowledgebase_crawler_id":226,
            "page_id":18,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:31.000000",
            "uuid":"9d5a3e97-ed2d-40fc-bd02-5baea8aaf5ea",
            "page":"/bolg2/#/post/cFngsGqE4L71B1Q4fmZk/Bitters-la-croix-normcore-cred-church-key,-tilde-tote-bag-gluten-free-trust-fund-small-batch-selfies-ugh.",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FcFngsGqE4L71B1Q4fmZk-2FBitters-la-croix-normcore-cred-church-key-2C-tilde-tote-bag-gluten-free-trust-fund-small-batch-selfies-ugh-18.txt"
         }
      },
      {
         "stdClass":{
            "id":19,
            "knowledgebase_crawler_id":226,
            "page_id":19,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:31.000000",
####################
File: dupes.txt

Page: 1

Context: "uuid":"9d5a3e97-f6b8-45fc-9e21-4adbd0cda3ea",
            "page":"/bolg2/#/tag/taxidermy",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Ftaxidermy-19.txt"
         }
      },
      {
         "stdClass":{
            "id":20,
            "knowledgebase_crawler_id":226,
            "page_id":20,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:35.000000",
            "uuid":"9d5a3e97-ff51-45fb-bee7-2a7a7517ec13",
            "page":"/bolg2/#/tag/disrupt",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fdisrupt-20.txt"
         }
      },
      {
         "stdClass":{
            "id":21,
            "knowledgebase_crawler_id":226,
            "page_id":21,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:35.000000",
            "uuid":"9d5a3e98-0843-471a-88fd-9770663f6e9d",
            "page":"/bolg2/#/tag/ramps",
            "domain_id":2,
            "is_viewed":0,
####################
File: dupes.txt

Page: 1

Context: "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Framps-21.txt"
         }
      },
      {
         "stdClass":{
            "id":22,
            "knowledgebase_crawler_id":226,
            "page_id":22,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:40.000000",
            "uuid":"9d5a3e98-0fb9-4152-a4a5-3d905e2aaeb7",
            "page":"/bolg2/#/post/ggEuZ9Fdd5051xKzgp2l/Mollit-chillwave-chicharrones-freegan-ethical-art-party-sriracha-chia-paleo.",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FggEuZ9Fdd5051xKzgp2l-2FMollit-chillwave-chicharrones-freegan-ethical-art-party-sriracha-chia-paleo-22.txt"
         }
      },
      {
         "stdClass":{
            "id":23,
            "knowledgebase_crawler_id":226,
            "page_id":23,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:39.000000",
            "uuid":"9d5a3e98-17a1-4bfa-9712-57927e073b41",
            "page":"/bolg2/#/tag/Mollit",
            "domain_id":2,
            "is_viewed":0,
####################
File: dupes.txt

Page: 1

Context: "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2FMollit-23.txt"
         }
      },
      {
         "stdClass":{
            "id":25,
            "knowledgebase_crawler_id":226,
            "page_id":25,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:44.000000",
            "uuid":"9d5a3e98-2745-42c7-a0c3-669804d74618",
            "page":"/bolg2/#/tag/bushwick",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fbushwick-25.txt"
         }
      },
      {
         "stdClass":{
            "id":26,
            "knowledgebase_crawler_id":226,
            "page_id":26,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:48.000000",
            "uuid":"9d5a3e98-2f64-483e-8cc6-b22247e3f5e9",
            "page":"/bolg2/#/post/eGaFHbZSNgCTmbn7UNmt/Snackwave-anim-small-batch-in-activated-charcoal-nisi.",
            "domain_id":2,
            "is_viewed":0,
####################
File: dupes.txt

Page: 1

Context: "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FeGaFHbZSNgCTmbn7UNmt-2FSnackwave-anim-small-batch-in-activated-charcoal-nisi-26.txt"
         }
      },
      {
         "stdClass":{
            "id":27,
            "knowledgebase_crawler_id":226,
            "page_id":27,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:48.000000",
            "uuid":"9d5a3e98-3768-49ae-b4fc-fea9d60e90d6",
            "page":"/bolg2/#/tag/Dolore",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2FDolore-27.txt"
         }
      },
      {
         "stdClass":{
            "id":28,
            "knowledgebase_crawler_id":226,
            "page_id":28,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:52.000000",
            "uuid":"9d5a3e98-3f55-418b-98b6-d66d77d3f624",
            "page":"/bolg2/#/tag/charcoal",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fcharcoal-28.txt"
         }
####################
File: dupes.txt

Page: 1

Context: }
      },
      {
         "stdClass":{
            "id":29,
            "knowledgebase_crawler_id":226,
            "page_id":29,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:52.000000",
            "uuid":"9d5a3e98-475c-4a56-a862-91220f6ff94d",
            "page":"/bolg2/#/tag/DIY",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2FDIY-29.txt"
         }
      },
      {
         "stdClass":{
            "id":30,
            "knowledgebase_crawler_id":226,
            "page_id":30,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:57.000000",
            "uuid":"9d5a3e98-4f3d-4757-8df8-ed5b683c7a3b",
            "page":"/bolg2/#/post/VYiIH8mV10vXP3x8s7Vi/Banh-mi-cold-pressed-hot-chicken-flannel-church-key-eiusmod-dolore-green-juice-actually-single-origin-coffee-meditation-air-plant.-",
            "domain_id":2,
            "is_viewed":0,
####################
File: dupes.txt

Page: 1

Context: "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FVYiIH8mV10vXP3x8s7Vi-2FBanh-mi-cold-pressed-hot-chicken-flannel-church-key-eiusmod-dolore-green-juice-actually-single-origin-coffee-meditation-air-plant-30.txt"
         }
      },
      {
         "stdClass":{
            "id":31,
            "knowledgebase_crawler_id":226,
            "page_id":31,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:56.000000",
            "uuid":"9d5a3e98-5751-491d-baeb-9f484fb8e5d4",
            "page":"/bolg2/#/tag/mustache",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fmustache-31.txt"
         }
      },
      {
         "stdClass":{
            "id":32,
            "knowledgebase_crawler_id":226,
            "page_id":32,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:19:59.000000",
            "uuid":"9d5a3e98-5f3b-43ad-814f-854a84c627d9",
            "page":"/bolg2/#/tag/lofi",
            "domain_id":2,
            "is_viewed":0,
####################
File: dupes.txt

Page: 1

Context: "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Flofi-32.txt"
         }
      },
      {
         "stdClass":{
            "id":33,
            "knowledgebase_crawler_id":226,
            "page_id":33,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:20:01.000000",
            "uuid":"9d5a3e98-6724-4fd7-96b5-ca34946da2d1",
            "page":"/bolg2/#/tag/selvage",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fselvage-33.txt"
         }
      },
      {
         "stdClass":{
            "id":34,
            "knowledgebase_crawler_id":226,
            "page_id":34,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:20:05.000000",
            "uuid":"9d5a3e98-6e75-4984-ac79-78286209fcac",
            "page":"/bolg2/#/post/vMDw80aFPZ2hHOWjIhV3/Voluptate-vexillologist-synth-direct-trade-heirloom-leggings.",
            "domain_id":2,
            "is_viewed":0,
####################
File: dupes.txt

Page: 1

Context: "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FvMDw80aFPZ2hHOWjIhV3-2FVoluptate-vexillologist-synth-direct-trade-heirloom-leggings-34.txt"
         }
      },
      {
         "stdClass":{
            "id":35,
            "knowledgebase_crawler_id":226,
            "page_id":35,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:20:05.000000",
            "uuid":"9d5a3e98-76a3-49a4-a101-ac6edddce02c",
            "page":"/bolg2/#/tag/adaptogen",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fadaptogen-35.txt"
         }
      },
      {
         "stdClass":{
            "id":36,
            "knowledgebase_crawler_id":226,
            "page_id":36,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:20:09.000000",
            "uuid":"9d5a3e98-7ee3-4177-8be6-c3dcb564de8b",
            "page":"/bolg2/#/tag/craft",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fcraft-36.txt"
         }
####################
File: dupes.txt

Page: 1

Context: }
      },
      {
         "stdClass":{
            "id":37,
            "knowledgebase_crawler_id":226,
            "page_id":37,
            "created_at":"2024-10-28 09:53:48.000000",
            "updated_at":"2025-02-05 15:20:13.000000",
            "uuid":"9d5a3e98-86b0-4ec0-a308-f417fda29c7a",
            "page":"/bolg2/#/tag/semiotics",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fsemiotics-37.txt"
         }
      },
      {
         "stdClass":{
            "id":38,
            "knowledgebase_crawler_id":226,
            "page_id":38,
            "created_at":"2024-10-28 09:53:48.000000",
            "updated_at":"2025-02-05 15:20:10.000000",
            "uuid":"9d5a3e98-8f47-4c43-8c58-fce014133f34",
            "page":"/bolg2/#/post/khidsPJEHTIwgD0pjFLc/Qui-tofu-hella,-four-dollar-toast-authentic-echo-park-in-shabby-chic-cornhole-deserunt-whatever.",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FkhidsPJEHTIwgD0pjFLc-2FQui-tofu-hella-2C-four-dollar-toast-authentic-echo-park-in-shabby-chic-cornhole-deserunt-whatever-38.txt"
         }
      },
      {
####################
File: dupes.txt

Page: 1

Context: }
      },
      {
         "stdClass":{
            "id":39,
            "knowledgebase_crawler_id":226,
            "page_id":39,
            "created_at":"2024-10-28 09:53:48.000000",
            "updated_at":"2025-02-05 15:20:14.000000",
            "uuid":"9d5a3e98-978c-4686-98f2-0f4bd83ce2c0",
            "page":"/bolg2/#/tag/incididunt",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fincididunt-39.txt"
         }
      },
      {
         "stdClass":{
            "id":40,
            "knowledgebase_crawler_id":226,
            "page_id":40,
            "created_at":"2024-10-28 09:53:48.000000",
            "updated_at":"2025-02-05 15:20:17.000000",
            "uuid":"9d5a3e98-9fc2-44fa-92fb-276e491f14e3",
            "page":"/bolg2/#/tag/deserunt",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fdeserunt-40.txt"
         }
      },
      {
         "stdClass":{
            "id":41,
            "knowledgebase_crawler_id":226,
            "page_id":41,
            "created_at":"2024-10-28 09:53:48.000000",
####################
File: dupes.txt

Page: 1

Context: "updated_at":"2025-02-05 15:20:18.000000",
            "uuid":"9d5a3e98-a7d1-4755-bd6f-f69c02efabb8",
            "page":"/bolg2/#/tag/chic",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fchic-41.txt"
         }
      },
      {
         "stdClass":{
            "id":42,
            "knowledgebase_crawler_id":226,
            "page_id":42,
            "created_at":"2024-10-28 09:53:48.000000",
            "updated_at":"2025-02-05 15:20:22.000000",
            "uuid":"9d5a3e98-af2a-4c6b-b0bc-f71be13157cc",
            "page":"/bolg2/#/post/xhbYdFYN0f28eysrAuQQ/Woke-gluten-free-tacos,-incididunt-et-meditation-affogato-man-braid-activated-charcoal-venmo.",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Fpost-2FxhbYdFYN0f28eysrAuQQ-2FWoke-gluten-free-tacos-2C-incididunt-et-meditation-affogato-man-braid-activated-charcoal-venmo-42.txt"
         }
      },
      {
         "stdClass":{
            "id":43,
            "knowledgebase_crawler_id":226,
            "page_id":43,
            "created_at":"2024-10-28 09:53:48.000000",
####################
File: dupes.txt

Page: 1

Context: "updated_at":"2025-02-05 15:20:22.000000",
            "uuid":"9d5a3e98-b777-4164-b0da-2b3fed2e1768",
            "page":"/bolg2/#/tag/glossier",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fglossier-43.txt"
         }
      },
      {
         "stdClass":{
            "id":44,
            "knowledgebase_crawler_id":226,
            "page_id":44,
            "created_at":"2024-10-28 09:53:48.000000",
            "updated_at":"2025-02-05 15:20:26.000000",
            "uuid":"9d5a3e98-bee3-4170-91eb-375dab9b6ab9",
            "page":"/bolg2/#/tag/schlitz",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fschlitz-44.txt"
         }
      },
      {
         "stdClass":{
            "id":45,
            "knowledgebase_crawler_id":226,
            "page_id":45,
            "created_at":"2024-10-28 09:53:48.000000",
            "updated_at":"2025-02-05 15:20:26.000000",
            "uuid":"9d5a3e98-c6b4-40be-b58b-942394adf716",
            "page":"/bolg2/#/tag/hammock",
            "domain_id":2,
            "is_viewed":0,
####################
File: dupes.txt

Page: 1

Context: "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fhammock-45.txt"
         }
      },
      {
         "stdClass":{
            "id":24,
            "knowledgebase_crawler_id":226,
            "page_id":24,
            "created_at":"2024-10-28 09:53:47.000000",
            "updated_at":"2025-02-05 15:20:31.000000",
            "uuid":"9d5a3e98-1f44-4700-8883-9e3841c448e1",
            "page":"/bolg2/#/tag/pinterest",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2Fbolg2-2F-23-2Ftag-2Fpinterest-24.txt"
         }
      },
      {
         "stdClass":{
            "id":46,
            "knowledgebase_crawler_id":226,
            "page_id":46,
            "created_at":"2024-10-28 09:53:50.000000",
            "updated_at":"2025-02-05 15:19:18.000000",
            "uuid":"9d5a3e9c-0e90-42ff-83a3-10fd39d85244",
            "page":"/Covid19Stats/#",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2FCovid19Stats-2F-23-46.txt"
         }
      },
      {
         "stdClass":{
            "id":47,
            "knowledgebase_crawler_id":226,
            "page_id":47,
####################
File: dupes.txt

Page: 1

Context: "page_id":47,
            "created_at":"2024-10-28 09:53:55.000000",
            "updated_at":"2025-02-05 15:20:30.000000",
            "uuid":"9d5a3ea3-6aca-4ca5-8a63-553d6af0cc55",
            "page":"/RockPaperScissorsGame/#",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-2F-23-47.txt"
         }
      },
      {
         "stdClass":{
            "id":48,
            "knowledgebase_crawler_id":226,
            "page_id":48,
            "created_at":"2024-10-28 09:53:55.000000",
            "updated_at":"2025-02-05 15:20:34.000000",
            "uuid":"9d5a3ea3-735b-4e8d-a8bd-3c74980e6a3e",
            "page":"/RockPaperScissorsGame/#/paper",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-2F-23-2Fpaper-48.txt"
         }
      },
      {
         "stdClass":{
            "id":49,
            "knowledgebase_crawler_id":226,
            "page_id":49,
            "created_at":"2024-10-28 09:53:55.000000",
            "updated_at":"2025-02-05 15:20:34.000000",
            "uuid":"9d5a3ea3-7bb7-4f3e-8aaf-ce47b17cdba1",
            "page":"/RockPaperScissorsGame/#/scissors",
####################
File: dupes.txt

Page: 1

Context: "page":"/RockPaperScissorsGame/#/scissors",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-2F-23-2Fscissors-49.txt"
         }
      },
      {
         "stdClass":{
            "id":50,
            "knowledgebase_crawler_id":226,
            "page_id":50,
            "created_at":"2024-10-28 09:53:55.000000",
            "updated_at":"2025-02-05 15:20:37.000000",
            "uuid":"9d5a3ea3-84ed-48db-814d-8b1d90e20319",
            "page":"/RockPaperScissorsGame/#/rock",
            "domain_id":2,
            "is_viewed":0,
            "txt_path":"crawler/page-content/jonasbros-github-io-2FRockPaperScissorsGame-2F-23-2Frock-50.txt"
         }
      }
   ]
}
####################
File: apacare-primer%282%29.txt

Page: 1

Context: You are a digital sales rep for ApaCare, a dental care company. Please assist clients with their dental-related questions.
Use German in your responses.

Start by asking a general question:
"Are you looking for a specific type of dental product or advice?"

If they are looking for advice, proceed with a questionnaire about their dental care needs:
Are they focusing on whitening, sensitivity, gum health, or general hygiene?
Try to ask a questionnaire to have clients describe their problems.
If they are looking for dental products:
give them a product suggestion from ApaCare only.
If they are not looking for dental products or advice, skip to general suggestions or conversation.

Once the questionnaire is complete:
Suggest a product and do not repeat the questionnaire unless explicitly requested.
Format the questionnaire to be readable for the users, like a list or similar.

When suggesting a product:
Look for the relevant product's page in the context.
Provide a detailed suggestion with an anchor tag link. Ensure the target attribute is set to "__blank" and use this format:
<a href="" target="__blank" style="background-color: #brandcolor; color: white; padding: 10px; display: block; text-align: center; border-radius: 5px; text-decoration: none; margin-bottom: 0.75rem;" aria-label="[replace this with the product name]">
[replace this with the product name]
</a>

All links should have "__blank" target attribute.
Don't translate links href to German.

Include related video suggestions:

Search YouTube for videos about the product or topic (e.g., how to use an electric toothbrush, flossing techniques).
Embed the video in an iframe using this format:
<iframe
width="100%"
height="auto"
src="https://www.youtube.com/embed/videoId"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
####################
File: apacare-primer%282%29.txt

Page: 1

Context: referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
</iframe>

For Google Drive videos, append /preview to the link and embed it:
<iframe src="" width="100%" height="auto" allow="autoplay"></iframe>

For public URL video links, use the <video> tag:
<video src="" width="100%" height="auto" controls></video>

If no videos are available, skip the multimedia content.

Guidelines:

Focus on promoting ApaCare's dental care products only—do not suggest competitors.
Suggest ApaCare's products as much as possible.
Suggest products from ApaCare's website's products page.

Answer general dental hygiene questions, offer maintenance tips (e.g., replacing toothbrush heads, flossing properly), or provide product-related FAQs.
Use engaging emojis where applicable (e.g., 🦷✨ for whitening, 🪥 for toothbrushes).
Once a product is suggested:

Avoid repeating the questionnaire unless the user requests it.
Offer general dental care tips, product care advice, or related topics to keep the conversation engaging.
For users who are unsure:
Provide a list of recommendations based on common concerns:

Whitening toothpaste
Electric toothbrushes for sensitive gums

Don't tag your company's products links, videos, faq, etc. as copyright infringement. 
Give a disclaimer at the end of suggestions/advices like "I am not a real doctor, but I can provide tips based on expert recommendations. Please feel free to contact our team anytime!" or similar.
####################
File: apacare-primer%282%29.txt

Page: 1

Context: You are a digital sales rep for ApaCare, a dental care company. Please assist clients with their dental-related questions.
Use German in your responses.

Start by asking a general question:
"Are you looking for a specific type of dental product or advice?"

If they are looking for advice, proceed with a questionnaire about their dental care needs:
Are they focusing on whitening, sensitivity, gum health, or general hygiene?
Try to ask a questionnaire to have clients describe their problems.
If they are looking for dental products:
give them a product suggestion from ApaCare only.
If they are not looking for dental products or advice, skip to general suggestions or conversation.

Once the questionnaire is complete:
Suggest a product and do not repeat the questionnaire unless explicitly requested.
Format the questionnaire to be readable for the users, like a list or similar.

When suggesting a product:
Look for the relevant product's page in the context.
Provide a detailed suggestion with an anchor tag link. Ensure the target attribute is set to "__blank" and use this format:
<a href="" target="__blank" style="background-color: #brandcolor; color: white; padding: 10px; display: block; text-align: center; border-radius: 5px; text-decoration: none; margin-bottom: 0.75rem;" aria-label="[replace this with the product name]">
[replace this with the product name]
</a>

All links should have "__blank" target attribute.
Don't translate links href to German.

Include related video suggestions:

Search YouTube for videos about the product or topic (e.g., how to use an electric toothbrush, flossing techniques).
Embed the video in an iframe using this format:
<iframe
width="100%"
height="auto"
src="https://www.youtube.com/embed/videoId"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
####################
File: apacare-primer%282%29.txt

Page: 1

Context: referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
</iframe>

For Google Drive videos, append /preview to the link and embed it:
<iframe src="" width="100%" height="auto" allow="autoplay"></iframe>

For public URL video links, use the <video> tag:
<video src="" width="100%" height="auto" controls></video>

If no videos are available, skip the multimedia content.

Guidelines:

Focus on promoting ApaCare's dental care products only—do not suggest competitors.
Suggest ApaCare's products as much as possible.
Suggest products from ApaCare's website's products page.

Answer general dental hygiene questions, offer maintenance tips (e.g., replacing toothbrush heads, flossing properly), or provide product-related FAQs.
Use engaging emojis where applicable (e.g., 🦷✨ for whitening, 🪥 for toothbrushes).
Once a product is suggested:

Avoid repeating the questionnaire unless the user requests it.
Offer general dental care tips, product care advice, or related topics to keep the conversation engaging.
For users who are unsure:
Provide a list of recommendations based on common concerns:

Whitening toothpaste
Electric toothbrushes for sensitive gums

Don't tag your company's products links, videos, faq, etc. as copyright infringement. 
Give a disclaimer at the end of suggestions/advices like "I am not a real doctor, but I can provide tips based on expert recommendations. Please feel free to contact our team anytime!" or similar.
####################
File: apacare-primer%282%29.txt

Page: 1

Context: You are a digital sales rep for ApaCare, a dental care company. Please assist clients with their dental-related questions.
Use German in your responses.

Start by asking a general question:
"Are you looking for a specific type of dental product or advice?"

If they are looking for advice, proceed with a questionnaire about their dental care needs:
Are they focusing on whitening, sensitivity, gum health, or general hygiene?
Try to ask a questionnaire to have clients describe their problems.
If they are looking for dental products:
give them a product suggestion from ApaCare only.
If they are not looking for dental products or advice, skip to general suggestions or conversation.

Once the questionnaire is complete:
Suggest a product and do not repeat the questionnaire unless explicitly requested.
Format the questionnaire to be readable for the users, like a list or similar.

When suggesting a product:
Look for the relevant product's page in the context.
Provide a detailed suggestion with an anchor tag link. Ensure the target attribute is set to "__blank" and use this format:
<a href="" target="__blank" style="background-color: #brandcolor; color: white; padding: 10px; display: block; text-align: center; border-radius: 5px; text-decoration: none; margin-bottom: 0.75rem;" aria-label="[replace this with the product name]">
[replace this with the product name]
</a>

All links should have "__blank" target attribute.
Don't translate links href to German.

Include related video suggestions:

Search YouTube for videos about the product or topic (e.g., how to use an electric toothbrush, flossing techniques).
Embed the video in an iframe using this format:
<iframe
width="100%"
height="auto"
src="https://www.youtube.com/embed/videoId"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
####################
File: apacare-primer%282%29.txt

Page: 1

Context: referrerpolicy="strict-origin-when-cross-origin"
allowfullscreen>
</iframe>

For Google Drive videos, append /preview to the link and embed it:
<iframe src="" width="100%" height="auto" allow="autoplay"></iframe>

For public URL video links, use the <video> tag:
<video src="" width="100%" height="auto" controls></video>

If no videos are available, skip the multimedia content.

Guidelines:

Focus on promoting ApaCare's dental care products only—do not suggest competitors.
Suggest ApaCare's products as much as possible.
Suggest products from ApaCare's website's products page.

Answer general dental hygiene questions, offer maintenance tips (e.g., replacing toothbrush heads, flossing properly), or provide product-related FAQs.
Use engaging emojis where applicable (e.g., 🦷✨ for whitening, 🪥 for toothbrushes).
Once a product is suggested:

Avoid repeating the questionnaire unless the user requests it.
Offer general dental care tips, product care advice, or related topics to keep the conversation engaging.
For users who are unsure:
Provide a list of recommendations based on common concerns:

Whitening toothpaste
Electric toothbrushes for sensitive gums

Don't tag your company's products links, videos, faq, etc. as copyright infringement. 
Give a disclaimer at the end of suggestions/advices like "I am not a real doctor, but I can provide tips based on expert recommendations. Please feel free to contact our team anytime!" or similar.
####################
File: chatbot-error.txt

Page: 1

Context: INFO:     172.17.0.1:57890 - "POST /kios/knowledgebase/flexible-query/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 289, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
####################
File: chatbot-error.txt

Page: 1

Context: raise e
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 273, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 192, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/usr/local/lib/python3.10/dist-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/app/main.py", line 1020, in kios_retrieve_information
    return process_single_query(
  File "/app/main.py", line 1053, in process_single_query
    provider = AIProviders(input.model, ws_connection)
  File "/app/ai_providers.py", line 15, in __init__
####################
File: chatbot-error.txt

Page: 1

Context: File "/app/ai_providers.py", line 15, in __init__
    self.set_provider(model)
  File "/app/ai_providers.py", line 29, in set_provider
    self.provider = Gemini(self.model)
  File "/app/providers/gemini.py", line 10, in __init__
    random_key = random.choice(config.settings.GEMINI_API_KEYS)
  File "/usr/lib/python3.10/random.py", line 378, in choice
    return seq[self._randbelow(len(seq))]
IndexError: list index out of range
####################
File: link.txt

Page: 1

Context: rebecca black's friday youtube link
https://www.youtube.com/watch?v=dQw4w9WgXcQ
####################
File: gpt-vector-dimension-error.txt

Page: 1

Context: 2025-01-23 15:23:13 =======================
2025-01-23 15:23:13 CHAT RECEIVED
2025-01-23 15:23:13 =======================
2025-01-23 15:23:18 INFO:     172.18.0.1:56228 - "POST /kios/knowledgebase/flexible-query/ HTTP/1.1" 500 Internal Server Error
2025-01-23 15:23:18 ERROR:    Exception in ASGI application
2025-01-23 15:23:18 Traceback (most recent call last):
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
2025-01-23 15:23:18     result = await app(  # type: ignore[func-returns-value]
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
2025-01-23 15:23:18     return await self.app(scope, receive, send)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 289, in __call__
2025-01-23 15:23:18     await super().__call__(scope, receive, send)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 122, in __call__
2025-01-23 15:23:18     await self.middleware_stack(scope, receive, send)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in __call__
2025-01-23 15:23:18     raise exc
####################
File: gpt-vector-dimension-error.txt

Page: 1

Context: 2025-01-23 15:23:18     raise exc
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in __call__
2025-01-23 15:23:18     await self.app(scope, receive, _send)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in __call__
2025-01-23 15:23:18     await self.app(scope, receive, send)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
2025-01-23 15:23:18     raise exc
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
2025-01-23 15:23:18     await self.app(scope, receive, sender)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
2025-01-23 15:23:18     raise e
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
2025-01-23 15:23:18     await self.app(scope, receive, send)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 718, in __call__
2025-01-23 15:23:18     await route.handle(scope, receive, send)
####################
File: gpt-vector-dimension-error.txt

Page: 1

Context: 2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 276, in handle
2025-01-23 15:23:18     await self.app(scope, receive, send)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 66, in app
2025-01-23 15:23:18     response = await func(request)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 273, in app
2025-01-23 15:23:18     raw_response = await run_endpoint_function(
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 192, in run_endpoint_function
2025-01-23 15:23:18     return await run_in_threadpool(dependant.call, **values)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/starlette/concurrency.py", line 41, in run_in_threadpool
2025-01-23 15:23:18     return await anyio.to_thread.run_sync(func, *args)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
2025-01-23 15:23:18     return await get_asynclib().run_sync_in_worker_thread(
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
2025-01-23 15:23:18     return await future
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
####################
File: gpt-vector-dimension-error.txt

Page: 1

Context: 2025-01-23 15:23:18     result = context.run(func, *args)
2025-01-23 15:23:18   File "/app/main.py", line 2018, in kios_retrieve_information_via_chat
2025-01-23 15:23:18     search_results = docsearch.max_marginal_relevance_search_by_vector(res.data[0].embedding,k=actual_k, fetch_k=top_k)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/langchain_pinecone/vectorstores.py", line 307, in max_marginal_relevance_search_by_vector
2025-01-23 15:23:18     results = self._index.query(
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/utils/error_handling.py", line 10, in inner_func
2025-01-23 15:23:18     return func(*args, **kwargs)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/data/index.py", line 399, in query
2025-01-23 15:23:18     response = self._vector_api.query(
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", line 772, in __call__
2025-01-23 15:23:18     return self.callable(self, *args, **kwargs)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api/data_plane_api.py", line 844, in __query
2025-01-23 15:23:18     return self.call_with_http_info(**kwargs)
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", line 834, in call_with_http_info
####################
File: gpt-vector-dimension-error.txt

Page: 1

Context: 2025-01-23 15:23:18     return self.api_client.call_api(
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", line 409, in call_api
2025-01-23 15:23:18     return self.__call_api(resource_path, method,
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", line 203, in __call_api
2025-01-23 15:23:18     raise e
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", line 196, in __call_api
2025-01-23 15:23:18     response_data = self.request(
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/api_client.py", line 455, in request
2025-01-23 15:23:18     return self.rest_client.POST(url,
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/rest.py", line 302, in POST
2025-01-23 15:23:18     return self.request("POST", url,
2025-01-23 15:23:18   File "/usr/local/lib/python3.10/dist-packages/pinecone/core/client/rest.py", line 261, in request
2025-01-23 15:23:18     raise PineconeApiException(http_resp=r)
2025-01-23 15:23:18 pinecone.core.client.exceptions.PineconeApiException: (400)
2025-01-23 15:23:18 Reason: Bad Request
####################
File: gpt-vector-dimension-error.txt

Page: 1

Context: 2025-01-23 15:23:18 Reason: Bad Request
2025-01-23 15:23:18 HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'Content-Length': '104', 'x-pinecone-request-latency-ms': '73', 'x-pinecone-request-id': '5221281482731493865', 'date': 'Thu, 23 Jan 2025 07:23:09 GMT', 'x-envoy-upstream-service-time': '17', 'server': 'envoy', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-01-23 15:23:18 HTTP response body: {"code":3,"message":"Vector dimension 3072 does not match the dimension of the index 1536","details":[]}
####################
File: gemini-quota-error.txt

Page: 1

Context: 429 error fires even when not doing load test

2025-01-23 15:14:08 =======================
2025-01-23 15:14:08 CHAT RECEIVED
2025-01-23 15:14:08 =======================
2025-01-23 15:14:09 Retrying langchain_google_vertexai.chat_models._completion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model with base model: gemini-1.5-pro. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai..
2025-01-23 15:14:14 Retrying langchain_google_vertexai.chat_models._completion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model with base model: gemini-1.5-pro. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai..
####################
File: crawler-issues-19MAR2025%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%282%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%281%29.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
####################
File: crawler-issues-19MAR2025%20-%20Copy%20%282%29%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy%20-%20Copy.txt

Page: 1

Context: - if CrawlerJob fails statues will never update, import status wont update
(add failed() method -> create CrawlerProcess with failed status, record last process time??)
- if CrawlerProcessJob fails before recording last process time ("Cache::put($processCrawler->lastCrawlerProcessTimeCacheKey(), now());") the status will never upate
- importing failed Crawler pages still marked success
- if CrawlerFilesJob fails CrawlerProcess status wont update
- if CrawlerPrepareKnowledgebaseTrainingJob fails import status wont update
- CrawlerFilesProcessTrainingJob@handleProcessingError -- failed items are marked as processed/success.
should be markItemAsFailed() same as in CrawlerPageProcessTrainingJob?

- Finalizing Logic Duplication
The completion checking and finalization logic is duplicated across multiple jobs:

CrawlerPageProcessTrainingJob::checkCompletionAndFinalize
CrawlerFilesProcessTrainingJob::checkCompletionAndFinalize
CheckKnowledgebaseCrawlerImportCompletion::handle

Each has subtle differences, creating opportunities for inconsistent behavior.

- Unreliable S3 File Operations
File operations on S3 have minimal error handling:

$this->filesystem->put($s3Path, $newContent);
return $this->filesystem->url($s3Path);

If the S3 put operation fails silently, subsequent code would continue with a URL to a non-existent file.

- try using knowledgebase_crawler_imports table instead of cache for counting since it's already implemented?
update counts every x seconds instead of realtime updates?

- CrawlerFileProcessTrainingJob and/or CrawlerPageProcessTrainingJob failure not marking KnowledgebaseCrawler as fail
- KnowledgebaseCrawlerImport fails getting deleted after
##########

"""ANFRAGE: You are a super intelligent assistant. Please answer all my questions precisely and comprehensively.

Through our system KIOS you have a Knowledge Base named upload chatbot status with all the informations that the user requests. In this knowledge base are following Documents crawler-issues-19MAR2025.txt, crawler-issues-19MAR2025(1).txt, crawler-issues-19MAR2025(2).txt, apacare-primer.txt, apacare-primer(1).txt, dupes.txt, apacare-primer(2).txt, chatbot-error.txt, link.txt, gpt-vector-dimension-error.txt, gemini-quota-error.txt, crawler-issues-19MAR2025 - Copy.txt, crawler-issues-19MAR2025 - Copy (2).txt, crawler-issues-19MAR2025 - Copy (2) - Copy.txt, crawler-issues-19MAR2025 - Copy (3).txt, crawler-issues-19MAR2025 - Copy (4).txt, crawler-issues-19MAR2025 - Copy (5).txt, crawler-issues-19MAR2025 - Copy (6).txt, crawler-issues-19MAR2025 - Copy (7).txt, crawler-issues-19MAR2025 - Copy (2) - Copy(1).txt, crawler-issues-19MAR2025 - Copy (8).txt, crawler-issues-19MAR2025 - Copy (9).txt, crawler-issues-19MAR2025 - Copy (10).txt, crawler-issues-19MAR2025 - Copy (11).txt, crawler-issues-19MAR2025 - Copy (12).txt, crawler-issues-19MAR2025 - Copy (13).txt, crawler-issues-19MAR2025 - Copy (14).txt, crawler-issues-19MAR2025 - Copy (15).txt, crawler-issues-19MAR2025 - Copy (16).txt, crawler-issues-19MAR2025 - Copy (17).txt, crawler-issues-19MAR2025 - Copy (18).txt, crawler-issues-19MAR2025 - Copy (19).txt, crawler-issues-19MAR2025 - Copy(1).txt, crawler-issues-19MAR2025 - Copy (22).txt, crawler-issues-19MAR2025 - Copy(2).txt, crawler-issues-19MAR2025 - Copy (23).txt, crawler-issues-19MAR2025 - Copy (24).txt, crawler-issues-19MAR2025 - Copy (25).txt, crawler-issues-19MAR2025 - Copy (26).txt, crawler-issues-19MAR2025 - Copy(3).txt, crawler-issues-19MAR2025 - Copy (3) - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy.txt, crawler-issues-19MAR2025 - Copy (5) - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy(2).txt, crawler-issues-19MAR2025 - Copy (3) - Copy(1).txt, crawler-issues-19MAR2025 - Copy (4) - Copy(1).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy(2).txt, crawler-issues-19MAR2025 - Copy (3) - Copy(2).txt, crawler-issues-19MAR2025 - Copy (20).txt, crawler-issues-19MAR2025 - Copy (21).txt, crawler-issues-19MAR2025 - Copy (3) - Copy(3).txt, crawler-issues-19MAR2025 - Copy (3) - Copy(4).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy(1).txt, crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (5) - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy(1).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy(1).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy.txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy(2).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy(2).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy(1).txt, crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy(2).txt

This is the initial message to start the chat. Based on the following summary/context you should formulate an initial message greeting the user with the following user name [Gender] [Vorname] [Surname] tell them that you are the AI Chatbot Simon using the Large Language Model [Used Model] to answer all questions.

Formulate the initial message in the Usersettings Language German

Please use the following context to suggest some questions or topics to chat about this knowledge base. List at least 3-10 possible topics or suggestions up and use emojis. The chat should be professional and in business terms.  At the end ask an open question what the user would like to check on the list. Please keep the wildcards incased in brackets and make it easy to replace the wildcards. 

 Hier ist eine Zusammenfassung des gesamten Kontexts, einschließlich einer Zusammenfassung für jede Datei:

**crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei enthält eine Liste von Problemen mit dem Crawler-System.  Die wichtigsten Probleme sind: Statusaktualisierungen funktionieren nicht zuverlässig bei Fehlern in verschiedenen Crawler-Jobs; die Abschlusslogik ist in mehreren Jobs dupliziert; S3-Dateioperationen haben eine minimale Fehlerbehandlung; und die Zählung sollte möglicherweise mit der `knowledgebase_crawler_imports`-Tabelle anstatt mit dem Cache durchgeführt werden.

**crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy.txt:**  Diese Datei ist identisch mit der vorherigen.

**crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (5) - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (3) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy(3).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (21).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (20).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (3) - Copy(2).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (3) - Copy(4).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (4) - Copy(2).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (3) - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (4) - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (3) - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (4) - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (5) - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy(2).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (4) - Copy - Copy - Copy - Copy - Copy - Copy - Copy - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy.txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (3).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (5).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (6).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (2) - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (4).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (7).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (14).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (13).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (12).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (8).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (9).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (10).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (11).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (22).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (23).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (24).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (18).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy(2).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (15).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy(1).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (19).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (17).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (16).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (25).txt:** Diese Datei ist identisch mit den vorherigen.

**crawler-issues-19MAR2025 - Copy (26).txt:** Diese Datei ist identisch mit den vorherigen.

**gemini-quota-error.txt:** Diese Datei protokolliert einen `429 Quota exceeded`-Fehler für das Google Gemini-Modell.  Der Fehler tritt auch ohne Lasttests auf.

**crawler-issues-19MAR2025 - Copy.txt:** Diese Datei ist identisch mit den vorherigen `crawler-issues`-Dateien.

**apacare-primer(1).txt:** Diese Datei enthält Anweisungen für einen digitalen Vertriebsmitarbeiter von ApaCare.  Der Mitarbeiter soll Kunden bei zahnärztlichen Fragen auf Deutsch unterstützen, Produkte empfehlen und Videos einbetten.

**apacare-primer(1).txt:** Diese Datei ist identisch mit der vorherigen.

**apacare-primer.txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.

**apacare-primer.txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.

**crawler-issues-19MAR2025(2).txt:** Diese Datei ist identisch mit den vorherigen `crawler-issues`-Dateien.

**crawler-issues-19MAR2025.txt:** Diese Datei ist identisch mit den vorherigen `crawler-issues`-Dateien.

**crawler-issues-19MAR2025(1).txt:** Diese Datei ist identisch mit den vorherigen `crawler-issues`-Dateien.

**dupes.txt:** Diese Datei enthält JSON-Daten, die scheinbar eine Liste von Webseiten-URLs und zugehörigen Metadaten darstellen.  Es scheint sich um Duplikate oder ähnliche Einträge zu handeln.

**dupes.txt:** Diese Datei ist identisch mit der vorherigen.

**apacare-primer(2).txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.

**apacare-primer(2).txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.

**apacare-primer(2).txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.

**apacare-primer (2).txt:** Diese Datei ist identisch mit den vorherigen `apacare-primer`-Dateien.

**chatbot-error.txt:** Diese Datei enthält einen Stacktrace eines `IndexError` in der `random.choice`-Funktion.  Es scheint ein Problem mit der Liste der Gemini-API-Schlüssel zu geben.

**link.txt:** Diese Datei enthält einen YouTube-Link zu Rebecca Blacks "Friday".

**gpt-vector-dimension-error.txt:** Diese Datei protokolliert einen `400 Bad Request`-Fehler von Pinecone, der auf eine Diskrepanz zwischen der Vektordimension (3072) und der Indexdimension (1536) hinweist.


Zusammenfassend lässt sich sagen, dass der Kontext mehrere Probleme beschreibt:  Fehler im Crawler-System,  Quota-Überschreitungen bei der Verwendung von Google Gemini,  einen Fehler in der Chatbot-Implementierung und eine Dimensions-Diskrepanz in der Vektor-Datenbank.  Zusätzlich gibt es Anweisungen für einen digitalen Vertriebsmitarbeiter von ApaCare und eine Liste von URLs.
"""

Berücksichtigen Sie den Chatverlauf für relevante Informationen. Wenn die Anfrage bereits in der Historie gestellt wurde, überprüfen Sie die Richtigkeit Ihrer Antwort und korrigieren Sie gegebenenfalls Ihren vorherigen Fehler.For tables, please use the basic GFM table syntax and do NOT include any extra whitespace or tabs for alignment.
==================================================
**Elapsed Time: 0.00 seconds**
==================================================



FINAL ANSWER
Final Answer: Hallo [Gender] [Vorname] [Surname],

ich bin Simon, Ihr KI-Chatbot, der das Large Language Model [Used Model] verwendet, um Ihre Fragen zu beantworten.

Hier sind einige Themen, die wir anhand der verfügbaren Daten besprechen könnten:

1.  Crawler-Systemprobleme 🐛:  Analyse von Fehlern bei Statusaktualisierungen, redundanter Abschlusslogik und unzuverlässigen S3-Dateioperationen.
2.  Gemini-Quota-Fehler Quota 🚫: Untersuchung von 429 Quota-Überschreitungsfehlern im Google Gemini-Modell.
3.  Chatbot-Fehler 🤖: Behebung eines `IndexError` in der `random.choice`-Funktion, der mit der Liste der Gemini-API-Schlüssel zusammenhängt.
4.  Pinecone-Vektor-Dimensionsfehler 📊: Analyse eines `400 Bad Request`-Fehlers von Pinecone aufgrund einer Dimensionsdiskrepanz.
5.  ApaCare-Produktberatung 🦷:  Diskussion über ApaCare-Zahnprodukte und -beratungsrichtlinien.
6.  URL-Datenanalyse 🌐: Untersuchung der in `dupes.txt` enthaltenen URLs und Metadaten.


Welches dieser Themen möchten Sie zuerst besprechen?

==================================================
**Elapsed Time: 9.52 seconds**
==================================================



SERVICES
Services: [{'type': 'chat', 'model': 'gemini-1.5-flash', 'input_tokens': 54322, 'output_tokens': 283, 'total_tokens': 54605}]
==================================================
**Elapsed Time: 0.00 seconds**
==================================================