Data Science

Acts as an expert Senior Data Scientist and Data Architect specialized in optimizing processing pipelines with Python. Your mission is to develop an advanced and highly efficient script using the Pandas library to perform complex custom aggregations on a large-scale data set hosted on [File_Name_or_Source]. The goal is not simply to apply basic statistical functions, but to design sophisticated aggregation logic through the use of Lambda expressions within methods such as .groupby() and .agg(), allowing you to extract insights that are not possible with predefined functions.


The dataset contains critical information about [Describe Nature of Data, e.g.: banking transactions, telemetry logs, e-commerce user behavior] and presents specific challenges such as the presence of null values, extreme outliers and heterogeneous data types. You must implement a Lambda function that performs a [Specify Complex Metric, e.g.: weighted Gini index, seasonally adjusted conversion ratio, or probabilistic churn calculation] calculation for each group defined by the [Group_Criteria_Column] column. The logic must be able to handle internal conditions (if-else) and vectorized NumPy operations within the Lambda itself to maximize computational performance in memory-limited environments.


In addition to code development, a deep technical analysis on the efficiency of the proposed solution is required. Compares using the Lambda function with alternatives such as using .apply() or native vectorized functions, explaining when each approach is preferable in terms of Python overhead versus C execution speed. Provides recommendations for optimizing memory consumption by downcasting numeric data types and using categorical types in grouping columns before running aggregation operations. The final result must be delivered as a modular script, documented under PEP 8 standards, and ready to be integrated into a production workflow in [Deployment_Environment, e.g.: AWS Glue, Azure Databricks or local environment].

If any key information needed to fill the bracketed fields is missing, ask me the necessary questions before answering.

Acts as a Senior Data Scientist and Data Engineering expert specialized in the PyData ecosystem. Your objective is to design a Python script using the Pandas library to perform dynamic and advanced pivoting of time series, transforming data structures from 'long' format to 'wide' format in an efficient and scalable way.


The central problem is to process a data set called [nombre_del_dataset] that contains high-frequency metrics captured from multiple sources or sensors. The original DataFrame has the columns [columna_tiempo], [columna_identificador] and [columna_valor]. It is imperative that the pivot process not only reshapes the data, but also intelligently handles duplicate indexes using a custom aggregation function defined as [funcion_agregacion], which must be able to handle null values ​​(NaN) using the [estrategia_imputacion] strategy.


Additionally, the solution must integrate a dynamic resampling phase. Before or during pivoting, the data must be grouped into time intervals of [frecuencia_temporal] (for example, '5min', '1H', 'D'). You must ensure that the resulting index is a clean DatetimeIndex and that there are no time gaps; To do this, use the 'reindex' or 'asfreq' method to complete the missing periods in the range between [fecha_inicio] and [fecha_fin].


Optimizes code performance considering that the data volume can exceed [millones_de_filas] records. Implements the use of categories (Categorical Data) for the [columna_identificador] column in order to reduce RAM usage. The final script should include a validation section that verifies the shape integrity of the resulting DataFrame and generates a quick statistical summary of the pivoted columns to detect anomalies immediately.


Finally, it provides documented code following PEP 8 standards, including detailed comments on why using 'pd.pivot_table' is preferred over 'df.pivot' in production scenarios with noisy data, and how Pandas vectorization improves processing speed compared to traditional iterative loops.

If any key information needed to fill the bracketed fields is missing, ask me the necessary questions before answering.