At Daltix (Full-time), in Lisbon, Portugal
Expires at: 2020-02-08
We are looking for talented profiles to help build and maintain the distributed data collection system that is at the heart of our business. You'll be in the front-lines, facing massive (but interesting!) challenges as we try to scrape all retail data available.
We are a data-driven company which collects and processes more than 600GB of raw data (HTML) daily. We leverage big data technologies such as Serverless, Spark on AWS EMR to crunch these volumes of data and make it queryable.
In this role, you will ensure that our data collection engine, which consists of distributed web crawlers, is state of the art and ahead of our competition. You will ensure that we can scrape any webshop, no matter the ban-detection that has been put in place. Then, it will be important that proper monitoring tools are in place. We are currently scraping 60 sites and your goal is to at least triple that without losing completeness and quality.
- Designing & developing internal tools and frameworks.
- Designing & developing tools to automate testing & QA.
- Building & maintaining distributed data collection systems on AWS.
- Guarantee data integrity and quality by extending our logging, monitoring and outlier detection systems.ng
- Knowledge of Computer Engineering (through formal education or other means)
- Good understanding REST API's, databases and SQL
- Linux command-line experience
- Highly proficient in spoken and written English
- You are passionate about software engineering
Nice to have
- Experience working on top of Amazon Web Services
- Excellent coding skills and deep knowledge of web technologies
- Excellent problem-solving capabilities and a critical mindset
- You get energy from working in a highly complex and challenging startup environment with a high tech product
- DevOps & automation (Terraform, Ansible)
Data analysis using Pandas (Python)
- Work with the latest tech stack
We're a quickly growing company => you're personal growth can be huge too.
Health Benefits - Comprehensive coverage for medical
Meal allowance - Monthly meal card along with a fully stocked kitchen with enough coffee and fruits, along with monthly team drinks and dinners
Work-Life Balance - We trust you to know your schedule and work when you feel most productive
Learning and Development - Attend meet-up, conferences, and events that interest you and benefit your personal and career growth.