In this tutorial we are going to be using C# to scrape Craigslist to look for web development gigs. We will be using .NET Core with ScrapySharp. To get started, you will need to install the .NET Core SDK, Visual Studio Code, and the C# extension for Visual Studio Code. We will be creating a console app, but you can take a similar approach in any other type of project.

Creating Our Console App

The first thing we'll need to do is create a folder for this project. You can put it wherever your like and name it anything you want. For this project, I created a folder called "Scraper" and placed it in my desktop. But, you an place this anywhere you'd like

Once you have that created, open that folder in VS Code. .NET Core comes with a command-line interface we will use to create our console app. In VS Code, open your terminal (Terminal → new Terminal) and run this command:

dotnet new console

After this command runs, you should have the following files:

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/8b3496c3-9532-4651-b80e-231999f8aa44/Screen_Shot_2020-01-24_at_7.29.47_AM.png

The main file we will be working out of is Program.cs and it should look like this:

using System;

namespace Scraper
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

Configuring Our Debug Settings

In order to debug the application, we need to configure a launch.json file. To do this, click on the debugger icon in VS Code. Then click on "create launch.json file".

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/500dacd3-42e9-4d30-af5f-0baa03b36cba/Debug_Settings.png

After you click on create launch.json file, you should be presented with a dropdown list. In that list, selected .Net Core. This will create this configuration in your launch.json file. Before we continue, we're going to change one property so we can capture inputs. Change the console property from internalConsole to externalTerminal.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: <https://go.microsoft.com/fwlink/?linkid=830387>
    "version": "0.2.0",
    "configurations": [
        {
            "name": ".NET Core Launch (console)",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "build",
            "program": "${workspaceFolder}/bin/Debug/netcoreapp2.1/Scraper.dll",
            "args": [],
            "cwd": "${workspaceFolder}",
            "console": "externalTerminal",
            "stopAtEntry": false
        },
        {
            "name": ".NET Core Attach",
            "type": "coreclr",
            "request": "attach",
            "processId": "${command:pickProcess}"
        }
    ]
}

To test that it is working, click on the play button in the debug window:

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/e5ea10eb-5b92-4c23-be7c-f50e3b17e25a/Screen_Shot_2020-01-21_at_7.39.31_AM.png

This will launch your terminal on Mac or command prompt on Windows.

Adding ScrapySharp