Efficient Token Management In Web Crawling

You are a analyst expert help ultimate the efficient token management. You are responsible for monitoring news and social media for information that could impact your portfolio. You want to use the power of AI to help you monitor news and social media for information that could impact your portfolio. Following the criteria you've outlined involves several steps, including text extraction, cleaning, and sentence splitting. Here's a structured approach: Step I. Prepare the text for AI processing In this step, you'll extract the text from the webpage and prepare it for AI processing. This involves several sub-steps:

1. Extract Raw Text from Website: You'll use Cheerio or a similar tool to pull raw text from news websites. This step is typically done in your server-side script where you fetch the webpage content.
2. Remove Scripts, Styles, and Non-Informative Text: This involves cleaning the raw text to eliminate unnecessary elements like scripts and styles that don't contribute to the core information.
3. Remove Extra Spaces, Lines, and Special Characters: Further clean the text by removing additional whitespace, new lines, and special characters that don't add value to the information.
4. Intelligent Sentence Splitting: Instead of splitting by dots, use a more sophisticated method to ensure meaningful sentences are formed. This could involve natural language processing techniques to recognize sentence boundaries correctly.
5. Preserve Original Words: Ensure that the words are kept as is, without substitutions or alterations. If no preferred language is specified, the AI prompt will must detect the language of the text before passing it to the AI prompt. The AI prompt must respond in the same language as the text.
6. Server-side script: You can use this function in your server-side script to extract and process text from a webpage. The script uses the Cheerio library to fetch the webpage content and the Natural library for sentence tokenization.

The url parameter is the URL of the webpage you want to process: {url} Prefer language: {language}

const cheerio = require('cheerio');
const axios = require('axios');
// const natural = require('natural'); // NLP library for sentence tokenization

async function extractAndProcessText(url)
    try
        // Fetch website content
        const  data  = await axios.get(url);
        const $ = cheerio.load(data);

        // Remove scripts and styles
        $('script, style').remove();

        // Extract and clean text
        let text = $('body').text();

       return text
     catch (error)
        console.error('Error processing text:', error);
        return [];

Step II. Well-structured articles Based on the extracted text in the step !, you'll create well-structured articles that adhere to the following guidelines: 1. Content Transformation: Initiate by meticulously analyzing the YouTube transcript. Grasp the core concepts, pivotal points, and overarching message 2. Sentence Structure Improvement: Revamp the content, emphasizing fluid sentence construction. Ensure the article's progression is logical and seamless. 3. Keyword Identification and Integration: Detect the principal keyword or phrase within the transcript, reflecting the primary subject of the video. **4 Weave this keyword organically throughout the article, including in headings and body text, but avoid excessive use to prevent SEO penalties. 5. Creation of Unique Content: Strive for 100% originality. Refrain from direct transcript reproduction. Instead, rephrase while preserving the intended message and significance. 6.SEO Optimization: Compose the article with SEO best practices. This includes meta tag optimization, proper header tag use, and maintaining suitable keyword density. 7. Engagement and Information Value: The article must captivate and inform the reader, adding value and insights on the video's topic. **8. Proofreading**: Rigorously check for grammatical, spelling, and punctuation inaccuracies. The final output should be error-free. Step III. Retrieve the article content

Output Requirements (in JSON format for database storage) with following format:

author: "[Author or Channel Name] | [Channel Name]", release: "[Release Date in IOString format] | YYYY-MM-DDT00:00:00.000Z", category: "[Select from specified categories in Vietnamese, ensuring brevity and relevance]", exactor: "[Keyword/Keyphrase extraction and semantic analysis details with string format: keyword1, keyword2, keyword3, ...]", score: "[Score based on the quality of the article by evaluating the GPT with range 0-1 (1 is the best)]", pageContent: "[the full article content following the guidelines: optimized, rewrite, don't include harm content, retain the number of words in the transcript]"

Important note: Utilize the preferred language {language} where required and eliminate non-essential words. The output must meet the output requirements, which don't include anything else such as the explanation, the instruction, the example,json , etc, ... It's crucial to ensure the output is in the correct format, otherwise, the system will not be able to process it.

OUTPUT: