State of South Carolina Government

Data Analytics and Systems Engineer - TBD

South Carolina Hybrid Full-time

The mission of the South Carolina Department of Education (SCDE) is to serve students, support teachers, empower parents, and engage the community so that every student graduates prepared to reach their full potential. About the Role SCDE is seeking a data engineer to support Education Scholarship Trust Fund (ESTF) modernization and the agency’s education data systems. The primary responsibility is to build reliable pipelines that connect governed source files, including vendor assessment files, the Ed-Fi API, and a Snowflake/dbt longitudinal data warehouse.

The engineer will use Earthmover, an open-source Python package, to transform vendor and other CSV files into Ed-Fi API-compliant JSON, and Lightbeam to validate and send those records to the API. The role includes writing, testing, and maintaining reusable assessment bundles that bring vendor assessment results into the statewide data system. Reports, dashboards, and visualizations are a required secondary responsibility once the pipeline is established.

This position requires familiarity with interoperable data ecosystems, a strong interest in connecting systems, and the ability to work with urgency. A working, tested, and documented end-to-end product is expected within eight months of starting. What You’ll Do Build and maintain data pipelines. Transform governed CSV and other tabular files into Ed-Fi-compliant JSON using Earthmover, and validate and publish records through Lightbeam.

Analyze vendor assessment files. Study each vendor’s file layout and data dictionary, decide which fields are required or optional, and build an anonymized sample file for testing. Meet with vendors and program offices to understand assessments and file changes. Model assessments in Ed-Fi. Decide how a vendor’s tests, sub-tests, scores, performance levels, grade levels, and subjects map to Ed-Fi resources and descriptors (Assessment, ObjectiveAssessment, StudentAssessment, StudentObjectiveAssessment).

Present the mapping for review under Ed-Fi assessment governance practices. Write assessment bundles. Start from the template bundle and build the full package: seed files for reference data such as descriptors and objective assessments; the earthmover.yaml configuration (sources, transformations such as joins, date conversion and derived IDs, and destinations); and the JSON templates that produce each Ed-Fi record. Use SQL, Jinja, and Python (and dbt where appropriate) to make transformations repeatable.

Match students correctly. Configure and test student ID crosswalking against the Ed-Fi roster, review unmatched-student reports, and work with districts and vendors to resolve ID problems. Run, test, and send the data. Run each bundle on sample and production-sized files, reconcile record counts, and spot-check values against the source file. Use Lightbeam to send records to the Ed-Fi API in dependency order (assessments first, student results last), manage API credentials securely, read rejection logs, find root causes, fix the bundle or data, and resend.

Keep bundles current. Update bundles when a vendor changes its file format, when assessments are added, and when the Ed-Fi data standard version changes. Support safe reruns. Document and share. Write the README, run instructions, parameters, and known issues for every bundle. Submit work for peer review through GitHub and contribute improvements the Ed-Fi community can reuse. Support the longitudinal warehouse. Develop and maintain Snowflake/dbt models and verify that API records flow correctly into downstream datasets.

Validate data across systems. Reconcile source files, API records, and warehouse outputs; resolve identifier mismatches, missing records, duplicates, and load errors. Deliver a working product within eight months. Set milestones, demonstrate progress, coordinate dependencies, and raise blockers early. Deliver a repeatable pipeline and an initial usable ESTF report or analytical view. Build reports and dashboards. Use integrated ESTF data to provide timely insights for state leadership, program analysts, and educational staff, consistent with source availability and refresh schedules.

Coordinate with program and technical staff. Work with assessment program staff, vendors, district data contacts, infrastructure teams, and implementation partners on schedules, file specifications, data requirements, business rules, acceptance criteria, and error resolution. Explain technical data decisions to non-technical staff and vendors. Use approved AI coding tools responsibly. Review, test, debug, and document generated code, and follow agency security, privacy, FERPA, and access-control requirements.

Communicate progress, track deliverables, and perform other duties as assigned. This position is located in the Office of Information Technology. Examples of Duties A bachelor's degree in computer science or related field and experience in computer system development and modification in either an on-premises, cloud-based, or hybrid environment. Relevant experience may substitute for the bachelor's degree on a year-for-year basis.

Qualifications Working knowledge of Python and SQL, including joins, grouping, and null handling, and the ability to read, test, debug, and maintain transformation code. Earthmover runs on a SQL engine, so SQL is the most transferable skill. Experience writing and debugging structured data transformations (SQL, Python, R, SAS, or an ETL tool) to join, aggregate, reshape, and clean tabular data from files you did not create.

Experience working with delimited files (CSV and similar), including inconsistent date formats, leading zeros in IDs, empty versus null values, and changes in vendor layouts. Experience transforming tabular data into JSON and working with REST APIs. Experience with the Ed-Fi Data Standard, ODS/API, or another standards-based education data model, including descriptors, namespaces, identifiers, and resource references.

Experience with ETL and modern ELT pipelines end-to-end, or with pipeline and orchestration tools such as dbt, Airflow, Earthmover, or Lightbeam. Background in K-12 assessment data (scale scores, standard errors, performance levels, sub-scores, grade levels, and subjects), ESTF, student information systems, or statewide education reporting. Experience with Git and GitHub: branching, pull requests, code review, and keeping sensitive data out of repositories.

Experience calling REST APIs with OAuth client credentials and diagnosing HTTP error responses. Experience with student-level data governed by the Family Educational Rights and Privacy Act (FERPA). Experience with automated testing, reconciliation, monitoring, and production pipeline support. Experience with Snowflake, dbt, and Jinja (or a similar templating language such as Liquid or Mustache). Experience with Power BI, Preset/Apache Superset, or comparable visualization tools.

Ability to read and write YAML configuration files (indentation-sensitive syntax, parameters, and environment variables) and JSON, and to reason about nested objects and arrays well enough to tell why a record is invalid. Comfort using a command line to run tools, pass parameters, and read logs and error output. Understanding of data pipelines, source-to-target mapping, validation, error handling, and repeatable processing.

Ability to document a data mapping so someone else can maintain it, explaining why a source field maps to a target field and not only what the mapping is. Familiarity with interoperable data ecosystems, templating tools such as Jinja, and Git-based version control. Ability to interpret data models and API documentation, troubleshoot across systems, and deliver a working solution within a defined timeline. Attention to detail: a single misspelled code value or misplaced comma can fail a whole load.

Strong analytical judgment and clear communication, including explaining technical data decisions to non-technical program staff and vendors; ability to learn a new open-source toolset mostly from documentation and working examples; and ability to work independently through ambiguous vendor specifications and know when to escalate. Experience may be demonstrated through professional work, academic projects, or substantive personal projects.

Candidates do not need prior experience with every tool listed. Strong data engineering fundamentals and the ability to learn and deliver are essential. Supplemental Information Skills You Will Develop Writing complete Earthmover bundles: sources, transformations, destinations, and project composition. Modeling a vendor assessment into Ed-Fi Assessment, ObjectiveAssessment, StudentAssessment, and StudentObjectiveAssessment resources.

Running Lightbeam and resolving rejected records. Applying Ed-Fi assessment data-governance conventions. How We Work We value practical solutions that connect systems and reduce manual work. The expectation is a tested, documented pipeline that staff can operate and maintain, not a one-time prototype. Approved AI tools may support development, but the engineer remains responsible for code quality, data accuracy, and security.

Application Prompt Describe a data pipeline or integration you built. Explain the source data, transformation logic, destination, your contribution, and how you tested and maintained it. Include any use of Python, SQL, APIs, dbt, Jinja, or AI-assisted development. Do not submit confidential data or proprietary code. As a part of a generous compensation package, we offer comprehensive benefits that include generous paid time off, preeminent health care benefits, professional development opportunities, pension, a 401(k), and a hybrid work schedule with opportunity for remote work.

The South Carolina Department of Education offers an exceptional benefits package that includes: Health, Dental, Vision, Long Term Disability, and Life Insurance for Employee, Spouse, and Children. 15 days annual (vacation) leave per year. 15 days sick leave per year. 13 paid holidays. State Retirement Plan and Deferred Compensation Programs. ONLY THOSE APPLICANTS CHOSEN FOR AN INTERVIEW WILL BE NOTIFIED BY LETTER THAT THE POSITION HAS BEEN FILLED. http://www.ed.sc.gov/jobs/human-resources/title-ix-nondiscrimination/

Free tool / Resume tailor

Tailor your resume for this job

Paste your resume and get 3 to 8 of your own lines rewritten in the words this posting uses, with the reason for each change. The job is already filled in, and you don't need an account.

Job description provided by State of South Carolina.
View the original posting