Puppeteer

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://www.google.com');
  await page.type('input[name="q"]', 'puppies');
  await page.click('input[type="submit"]');

  // Wait for search results to load
  await page.waitForSelector('#search');

  console.log('Search results loaded!');

  await browser.close();
})();

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch();

const page = await browser.newPage();

await page.goto('https://www.google.com');

await page.type('input[name="q"]', 'puppies');

await page.click('input[type="submit"]');

await page.waitForSelector('#search');

console.log('Search results loaded!');

await browser.close();

Puppeteer:

Here are some of its key features:

1. Automating user interactions: With Puppeteer, you can simulate user interactions with a web page, such as clicking buttons, filling out forms, and navigating to different pages.

2. Generating screenshots and PDFs: Puppeteer allows you to generate screenshots and PDFs of web pages, which can be useful for testing and debugging.

3. Web scraping: Puppeteer makes it easy to scrape data from web pages, allowing you to extract information like prices, product details, and more.

4. Performance testing: Puppeteer provides tools for measuring the performance of web pages, including metrics like page load time and resource usage.

5. Mobile emulation: Puppeteer can simulate mobile devices, allowing you to test how your web pages look and perform on different devices and screen sizes.

6. Headless mode: Puppeteer can run in headless mode, which means it runs without a visible user interface, making it faster and more efficient.

7. Easy setup: Puppeteer can be installed with npm and is easy to set up, making it accessible to developers of all skill levels.

Here are some of the key classes in Puppeteer:

1. Browser: The Browser class represents a browser instance, which can be used to create new pages and perform other browser-level tasks.

2. Page: The Page class represents a web page, and provides methods for interacting with the page, such as navigating to a URL, clicking elements, and filling out forms.

3. ElementHandle: The ElementHandle class represents a DOM element on a web page, and provides methods for interacting with the element, such as clicking it, typing into it, and getting its properties.

4. Frame: The Frame class represents a frame or iframe on a web page, and provides methods for interacting with the frame, such as navigating it and evaluating JavaScript in it.

5. Request: The Request class represents a network request made by a web page, and provides information about the request, such as its URL, headers, and response.

6. Response: The Response class represents a network response received by a web page, and provides information about the response, such as its status code, headers, and content.

Here are some of the key functionalities of Puppeteer:

1. Web page automation: With Puppeteer, you can automate interactions with web pages, such as clicking buttons, filling out forms, and navigating to different pages. This allows you to test and debug web pages, and automate repetitive tasks.

2. Web scraping: Puppeteer makes it easy to scrape data from web pages, allowing you to extract information like prices, product details, and more. This can be useful for a variety of applications, such as data mining and price comparison.

3. Performance testing: Puppeteer provides tools for measuring the performance of web pages, including metrics like page load time and resource usage. This allows you to optimize the performance of your web pages and ensure that they are fast and responsive.

4. PDF and screenshot generation: Puppeteer allows you to generate PDFs and screenshots of web pages, which can be useful for testing and debugging, as well as for generating reports and documentation.

5. Mobile emulation: Puppeteer can simulate mobile devices, allowing you to test how your web pages look and perform on different devices and screen sizes. This can be useful for ensuring that your web pages are responsive and mobile-friendly.

6. Headless mode: Puppeteer can run in headless mode, which means it runs without a visible user interface. This makes it faster and more efficient, and allows you to automate tasks without being distracted by a visual interface.

1. Mouse interactions: Puppeteer allows you to simulate mouse interactions with a web page using the mouse object. You can move the mouse to a specific point on the page using the move method, click an element using the click method, and perform other mouse actions using other methods like down, up, and wheel.

// Example: Click on a button using the mouse
await page.waitForSelector('#my-button');
const button = await page.$('#my-button');
await button.click();

2. Keyboard interactions: Puppeteer also allows you to simulate keyboard interactions with a web page using the keyboard object. You can type text into an element using the type method, press and release specific keys using the press and release methods, and more.

// Example: Type "hello world" into an input field using the keyboard
await page.waitForSelector('#my-input');
const input = await page.$('#my-input');
await input.type('hello world');

3. File chooser: Puppeteer provides a way to simulate the selection of a file using the FileChooser class. You can use the setFiles method to set the files to be uploaded, and then use the accept method to accept the file selection.

// Example: Upload a file using a file chooser
await page.waitForSelector('#my-file-input');
const input = await page.$('#my-file-input');
const [fileChooser] = await Promise.all([
  page.waitForFileChooser(),
  input.click(),
]);
await fileChooser.setFiles('/path/to/my/file.pdf');
await fileChooser.accept();

4. Browser context: Puppeteer allows you to create separate browser contexts using the BrowserContext class. A browser context is like a separate instance of the browser that has its own cookies, cache, and other state. This can be useful for testing scenarios where you need to isolate the state of the browser.

// Example: Create a new browser context and navigate to a page
const context = await browser.createIncognitoBrowserContext();
const page = await context.newPage();
await page.goto('https://www.example.com');

5. Page navigation: Puppeteer provides a variety of methods for navigating between pages and controlling the browser history. You can navigate to a new page using the goto method, go back or forward in the browser history using the goBack and goForward methods, and reload the current page using the reload method.

// Example: Navigate to a new page and go back in the browser history
await page.goto('https://www.example.com');
await page.goBack();

6. Element handling: Puppeteer provides a variety of methods for interacting with elements on a page, including selecting elements by CSS selector, XPath, or other criteria, getting the text or value of an element, and more.

// Example: Get the text of a paragraph element
await page.waitForSelector('p');
const paragraph = await page.$('p');
const text = await page.evaluate(element => element.textContent, paragraph);
console.log(text);

7. Network interception: Puppeteer allows you to intercept and modify network requests made by a page using the intercept method. You can use this to mock responses, block certain requests, or modify the request or response headers.

// Example: Intercept a network request and modify the response
await page.setRequestInterception(true);
page.on('request', request => {
  if (request.url().endsWith('.png')) {
    request.respond({
      content: 'image/png',
      body: Buffer.from('fake-image-data'),
    });
  } else {
    request.continue();
  }
});

8. Page events: Puppeteer provides a variety of events that you can listen for on a page, such as the load event, the dialog event (which is triggered when a JavaScript alert or confirmation dialog appears), and the console event (which is triggered when a page logs a message to the console).

// Example: Log console messages to the console
page.on('console', message => console.log(message.text()));

 

1. Navigation: You can navigate to a new page using the goto method, which takes a URL as its argument. You can also retrieve the current URL of a page using the url method.

// Navigate to a new page
await page.goto('https://www.example.com');

// Get the current URL
const currentUrl = await page.url();
console.log(currentUrl);

2. Manipulating URLs: Puppeteer provides the URL class, which allows you to manipulate URLs by adding or removing query parameters, fragments, and more. You can create a new URL instance by passing a URL string to its constructor.

// Create a new URL object
const url = new URL('https://www.example.com');

// Add a query parameter
url.searchParams.set('key', 'value');

// Remove a fragment
url.hash = '';

// Get the updated URL string
const updatedUrl = url.toString();
console.log(updatedUrl);

3. Retrieving information about URLs: You can use the parse method of the url module to parse a URL string and retrieve information about its components, such as the protocol, hostname, and port.

// Parse a URL string
const url = new URL('https://www.example.com/path/to/page?query=parameter');

// Get the protocol
console.log(url.protocol); // "https:"

// Get the hostname
console.log(url.hostname); // "www.example.com"

// Get the port (returns an empty string if the port is not specified)
console.log(url.port); // ""

4. Extracting URLs from a page: You can use Puppeteer to extract URLs from a page, for example by finding all the links on a page and retrieving their href attributes.

// Get all links on the page and extract their URLs
const links = await page.$$eval('a', elements => elements.map(element => element.href));
console.log(links);

5. Checking the URL of a page: You can use Puppeteer to check whether the URL of a page matches a certain pattern, for example to make sure that a redirect has taken you to the expected page.

// Navigate to a page and check its URL
await page.goto('https://www.example.com/redirect');
const currentUrl = await page.url();
if (currentUrl === 'https://www.example.com/expected-page') {
  console.log('Redirect succeeded!');
} else {
  console.log('Redirect failed: expected URL was', expectedUrl, 'but actual URL was', currentUrl);
}

6. Handling URL fragments: Puppeteer allows you to retrieve and manipulate the fragment (the part of a URL after the # symbol) using the hash property of the URL object.

// Navigate to a page and retrieve the fragment
await page.goto('https://www.example.com/page#fragment');
const url = new URL(await page.url());
const fragment = url.hash;
console.log(fragment);

// Modify the fragment and navigate to the updated URL
url.hash = 'new-fragment';
await page.goto(url.toString());

 

List of some common methods provided by Puppeteer:

1. browser.newPage(): Creates a new Page object in the current browser context.

2. page.goto(url[, options]): Navigates to the specified URL.

3. page.click(selector[, options]): Clicks the element specified by the given selector.

4. page.type(selector, text[, options]): Types the given text into the element specified by the given selector.

5. page.waitForSelector(selector[, options]): Waits for the element specified by the given selector to be added to the page.

6. page.waitForNavigation([options]): Waits for the page to navigate to a new URL.

7. page.screenshot([options]): Takes a screenshot of the current page and returns it as a PNG buffer.

8. page.evaluate(pageFunction[, ...args]): Executes the given function in the context of the page and returns its result.

9. page.$(selector): Finds the first element matching the given selector.

10. page.$$(selector): Finds all elements matching the given selector.

11. page.setContent(html[, options]): Sets the HTML content of the page.

12. page.goBack([options]): Navigates to the previous page in the history.

13. page.goForward([options]): Navigates to the next page in the history.

14. page.waitForTimeout(timeout): Waits for the specified amount of time (in milliseconds) before continuing.

15. page.waitForFunction(pageFunction[, options[, ...args]]): Waits for the given function to return a truthy value before continuing.

16. page.waitForNavigation([options]): Waits for the page to navigate to a new URL.

17. page.setViewport(viewport) Sets the size of the viewport for the page.

18. page.evaluateHandle(pageFunction[, ...args]): Executes the given function in the context of the page and returns a handle to its result.

19. page.addScriptTag(options): Adds a script tag to the page.

20. page.setRequestInterception(value): Enables or disables request interception for the page.

Comparison between Puppeteer and Selenium

const puppeteer = require('puppeteer');
(async function () {
    const browser = await puppeteer.launch();
    console.log("Launched");
    const page = await browser.newPage();
    await page.goto('https://www.google.com/');
    console.log("In Site");
    await page.screenshot({ path: './Demo2.png' });
    console.log("Captured");
    browser.close();
})();

1. We import the Puppeteer library using require.

2. Launch a new browser.

3. Open a new page (tab) inside that browser.

4. Navigate to the Wikipedia page.

5. Take a screenshot.

6. Close the browser.

1. waitUntil: This method specifies when the page.waitFor...() method should stop waiting. The available options are:
• load: Wait until the page is fully loaded (i.e., all resources like images, stylesheets, scripts, etc. have finished loading).
• domcontentloaded: Wait until the DOMContentLoaded event is fired (i.e., the HTML content has been parsed and rendered, but some resources may still be loading).
• networkidle0: Wait until there are no more than 0 network connections for at least 500ms (i.e., the page is considered fully loaded when there are no pending network requests).
• networkidle2: Wait until there are no more than 2 network connections for at least 500ms (i.e., the page is considered fully loaded when there are no pending network requests or when there are at most 2 network connections left, which may be useful for pages that load resources dynamically).

2. timeout: This method specifies the maximum amount of time (in milliseconds) to wait for the condition to be met before timing out and throwing an error.

3. visible: This method specifies whether to wait for an element to become visible on the page.

4. hidden: This method specifies whether to wait for an element to become hidden on the page.

5. selector: This method specifies the CSS selector of the element to wait for.

const puppeteer = require('puppeteer');
const xlsx = require('xlsx');

(async () => {
  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();
  await page.goto('https://appexchange.salesforce.com/consulting');

  // Scroll to the bottom of the page to load all data
  await autoScroll(page);

  // Collect the data and save it to an Excel file
  const data = await page.evaluate(() => {
    const rows = [];
    document.querySelectorAll('.appx-tile-content-el').forEach(span => {
      const cells = [span.innerText];
      rows.push(cells);
    });
    return rows;
  });
  const wb = xlsx.utils.book_new();
  const ws = xlsx.utils.aoa_to_sheet(data);
  xlsx.utils.book_append_sheet(wb, ws, 'Data');
  xlsx.writeFile(wb, 'data.xlsx');

  await browser.close();
})();

async function autoScroll(page) {
  await page.evaluate(async () => {
    await new Promise((resolve, reject) => {
      let totalHeight = 0;
      const distance = 100;
      const scrollInterval = setInterval(() => {
        const scrollHeight = document.body.scrollHeight;
        window.scrollBy(0, distance);
        totalHeight += distance;
        if (totalHeight >= scrollHeight) {
          clearInterval(scrollInterval);
          resolve();
        }
      }, 1000); // Scroll every 1 second
    });

    await new Promise(resolve => setTimeout(resolve, 2000)); // Wait for 2 seconds after scrolling

    // Click the "Load More" button repeatedly until it's no longer present
    while (document.querySelector('#appx-load-more-button-id')) {
      document.querySelector('#appx-load-more-button-id').click();
      await new Promise(resolve => setTimeout(resolve, 2000)); // Wait for 2 seconds after clicking
    }

    await new Promise(resolve => setTimeout(resolve, 2000)); // Wait for 2 seconds after clicking all "Load More" buttons
  });
}