Overview We are seeking a technically strong developer/data engineer to build systems for large-scale collection, analysis, classification, and monitoring of publicly available data. The role involves designing tools that can identify channels, extract metadata, analyse trends, detect behavioural patterns, and organise data into searchable datasets for commercial intelligence and analytics purposes. This is a hands-on engineering role requiring strong API, scraping, data processing, and automation experience. ________________________________________ Core Responsibilities • Build systems to collect and process video metadata at scale • Integrate with the Data API and alternative public data sources • Develop automated workflows for: o Platform discovery o keyword/topic mining o influencer identification o trend analysis o engagement analysis o comment extraction o metadata classification • Build pipelines to clean, normalise, and structure large datasets • Create systems to classify channels by: o niche o geography o language o engagement quality o audience patterns • Develop databases and indexing systems for fast querying • Implement anti-duplication and entity matching systems • Build monitoring tools for ongoing tracking of channels/videos • Create dashboards, exports, and reporting tools • Optimise collection systems for reliability and scale ________________________________________ Required Technical Skills Strong Programming Ability Candidate must be highly competent in: • Python (preferred) • Node.js / TypeScript • Go Python stack experience is strongly preferred. ________________________________________ Required API & Data Skills Strong experience with: • API v3 • quota management • pagination • channel/video/comment endpoints • search optimisation • rate limit handling Must understand: • API authentication • batching • retry logic • parallelisation ________________________________________ Web Scraping & Automation Experience with: • Playwright • Puppeteer • Selenium • BeautifulSoup • Scrapy Must understand: • dynamic content extraction • browser automation • proxy handling • anti-bot limitations • resilient scraping architectures ________________________________________ Data Engineering Skills Required experience with: • PostgreSQL • MySQL • MongoDB • Elasticsearch / OpenSearch Must be able to: • design schemas • optimise indexing • handle large datasets • create efficient query structures ________________________________________ Data Processing & Analytics Candidate should understand: • NLP basics • keyword extraction • sentiment analysis • topic clustering • tagging/classification systems • duplicate detection • statistical analysis Preferred: • experience using LLM APIs or AI classification systems ________________________________________ Infrastructure & DevOps Useful skills include: • Docker • Linux server administration • cloud infrastructure (AWS/GCP/Azure) • task queues • cron automation • distributed processing Preferred: • Airflow • Celery • Kafka • Redis ________________________________________ Frontend / Dashboard Skills (Preferred) Useful but not mandatory: • React • Next.js • dashboard development • charting/data visualisation ________________________________________ Candidate Profile Ideal candidate: • has built large-scale scraping or intelligence systems before • understands data reliability issues • can work independently • writes clean maintainable code • understands scaling and automation • can think analytically about datasets and patterns ________________________________________ Nice-to-Have Experience • Social media analytics • Influencer discovery systems • OSINT tools • Ad-tech or martech systems • Search/indexing platforms • AI-assisted classification systems • Large-scale crawler development ________________________________________ Deliverables Candidate should be capable of building: • automated collection systems • structured databases • monitoring pipelines • analytics dashboards • export/reporting tools • scalable infrastructure for ongoing data ingestion ________________________________________ Important Notes • System must comply with applicable laws and platform policies. • Focus is on analysis of publicly accessible information. • Reliability, scalability, and data quality are critical.
German Speaker for Long-Term Tenant Search Category: Bulk Marketing, German Translator, Internet Marketing, Lead Generation, Leads, Marketing, Property Management, Real Estate Budget: €250 - €750 EUR
Gather Premium Office Files - PPTX, DOCX, XLSX Category: Business Analysis, Data Collection, Finance, Financial Analysis, Financial Modeling, Financial Research, PDF, Research Budget: $10 - $30 USD
13-May-2026 10:04 GMT
Adjustable Bed Renders & Animations Category: 3D Animation, 3D Modelling, 3D Rendering, 3D Visualization, 3ds Max, After Effects, Blender, Cinema 4D, Interior Design, Post Production Budget: £10 - £15 GBP
Compile Basic Match Statistics Category: Data Analysis, Data Entry, Data Management, Data Processing, Data Visualization, Excel, Google Sheets, Web Scraping Budget: $15 - $25 USD
13-May-2026 09:57 GMT
Igbo Religious Text Proofreading Category: Editing, English (US) Translator, English Translation, Language Tutoring, Microsoft Word, Proofreading, Translation, Writing Budget: $70 - $100 USD
