# Base Documentation Scrapper

This project is responsible of for scrapping the Base Documentation from the html files.

## Libraries/Tools Used
- **BeautifulSoup**: This library is used for scrapping
- **Pandas**: This library is used for data processing and data transformation

## Getting Started

> **Important Note:** Before we start scrapping, create a directory called `data` in the root directory. Furthermore, create two sub-directories called `html_docs` and `target`

We can run this script by starting the the `Docu_Scrapper.ipynb` in a jupyter notebook.

The script assumes that a directory called `data` exists. It constitutes two sub-directories, `html_docs` and `target`. The `html_docs` directory contains all html files that have to be scrapped, where as,  `target` contains the final scrapped file. 

It is important to note that we have to **manually** store the required files in the `html_docs` sub-directory.

## Project Contents
This project contains the following files/directories:

- `Docu_Scrapper.ipynb`: This is the python notebook script that is used to scrap html files and save into a csv file.
- `data/html_docs`: This directory contains the selected files html files that need to be scrapped
- `data/target`: This directory contains the final csv file containing the merged data from the html files.

## Further Work
Once we have the final document in the `data/target` directory, we need to clean this document. This process is performed in the `BASE_Docu_Cleaner` project.