{ "cells": [ { "cell_type": "markdown", "id": "anexo-0", "metadata": {}, "source": [ "# Anexo técnico: leer y reproducir el benchmark parcial\n", "\n", "Este cuaderno verifica el inventario y presenta las tablas completas ya medidas. No vuelve a ajustar modelos al abrirlo. La reproducción desde cero está en `reproduce.py`, incluido en esta descarga: ejecuta calentamiento y tres procesos independientes secuenciales, compara sus resultados y calcula los intervalos predeclarados.\n", "\n", "El mínimo está en `minimo.ipynb`. Este anexo añade conceptos de inventario, cobertura, métricas, incertidumbre y procedencia; no representa el flujo mínimo del modelador.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "anexo-1", "metadata": { "execution": { "iopub.execute_input": "2026-10-19T15:29:41.263744Z", "iopub.status.busy": "2026-10-19T15:29:41.263744Z", "iopub.status.idle": "2026-10-19T15:29:42.996205Z", "shell.execute_reply": "2026-10-19T15:29:42.996205Z" } }, "outputs": [ { "data": { "text/plain": [ "{'total': 28, 'executed': 28, 'failed': 0, 'pending': 0}" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import json\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "from h10_benchmark.contract import verify_coverage, verify_sources\n", "\n", "verify_sources(Path(\".\"))\n", "rows = json.loads(Path(\"results/results.json\").read_text(encoding=\"utf-8\"))\n", "verify_coverage(rows)" ] }, { "cell_type": "markdown", "id": "anexo-2", "metadata": {}, "source": [ "## Todas las combinaciones admitidas\n", "\n", "Cuatro carteras por siete recetas: 28. German y Taiwan suplementario usan una evaluación exterior de grupos X exactos, 25% y semilla 20261016; el target y el ID se excluyen de X. Los sintéticos conservan OOT 2024Q2. Sólo Desarrollo ajusta; Holdout es diagnóstico. No se acredita independencia entre clientes. No se seleccionó ninguna receta por estos resultados.\n", "\n", "Las tablas mantienen fallos y pendientes en el denominador. LC, HC, PAKDD y R siguen pendientes de la capa C y no forman parte de las 28 combinaciones admitidas.\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "anexo-3", "metadata": { "execution": { "iopub.execute_input": "2026-10-19T15:29:42.996205Z", "iopub.status.busy": "2026-10-19T15:29:42.996205Z", "iopub.status.idle": "2026-10-19T15:29:43.009291Z", "shell.execute_reply": "2026-10-19T15:29:43.009291Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
caserecipestatuserror
0germanfabricaejecutadoNone
1germansin_stepwiseejecutadoNone
2germancinco_binsejecutadoNone
3germanmonotonia_autoejecutadoNone
4germanoptbinning_mipejecutadoNone
5germanscorecardpyejecutadoNone
6germanskorecard_mipejecutadoNone
7taiwanfabricaejecutadoNone
8taiwansin_stepwiseejecutadoNone
9taiwancinco_binsejecutadoNone
10taiwanmonotonia_autoejecutadoNone
11taiwanoptbinning_mipejecutadoNone
12taiwanscorecardpyejecutadoNone
13taiwanskorecard_mipejecutadoNone
14consumo_comportamientofabricaejecutadoNone
15consumo_comportamientosin_stepwiseejecutadoNone
16consumo_comportamientocinco_binsejecutadoNone
17consumo_comportamientomonotonia_autoejecutadoNone
18consumo_comportamientooptbinning_mipejecutadoNone
19consumo_comportamientoscorecardpyejecutadoNone
20consumo_comportamientoskorecard_mipejecutadoNone
21consumo_driftfabricaejecutadoNone
22consumo_driftsin_stepwiseejecutadoNone
23consumo_driftcinco_binsejecutadoNone
24consumo_driftmonotonia_autoejecutadoNone
25consumo_driftoptbinning_mipejecutadoNone
26consumo_driftscorecardpyejecutadoNone
27consumo_driftskorecard_mipejecutadoNone
\n", "
" ], "text/plain": [ " case recipe status error\n", "0 german fabrica ejecutado None\n", "1 german sin_stepwise ejecutado None\n", "2 german cinco_bins ejecutado None\n", "3 german monotonia_auto ejecutado None\n", "4 german optbinning_mip ejecutado None\n", "5 german scorecardpy ejecutado None\n", "6 german skorecard_mip ejecutado None\n", "7 taiwan fabrica ejecutado None\n", "8 taiwan sin_stepwise ejecutado None\n", "9 taiwan cinco_bins ejecutado None\n", "10 taiwan monotonia_auto ejecutado None\n", "11 taiwan optbinning_mip ejecutado None\n", "12 taiwan scorecardpy ejecutado None\n", "13 taiwan skorecard_mip ejecutado None\n", "14 consumo_comportamiento fabrica ejecutado None\n", "15 consumo_comportamiento sin_stepwise ejecutado None\n", "16 consumo_comportamiento cinco_bins ejecutado None\n", "17 consumo_comportamiento monotonia_auto ejecutado None\n", "18 consumo_comportamiento optbinning_mip ejecutado None\n", "19 consumo_comportamiento scorecardpy ejecutado None\n", "20 consumo_comportamiento skorecard_mip ejecutado None\n", "21 consumo_drift fabrica ejecutado None\n", "22 consumo_drift sin_stepwise ejecutado None\n", "23 consumo_drift cinco_bins ejecutado None\n", "24 consumo_drift monotonia_auto ejecutado None\n", "25 consumo_drift optbinning_mip ejecutado None\n", "26 consumo_drift scorecardpy ejecutado None\n", "27 consumo_drift skorecard_mip ejecutado None" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "coverage = pd.DataFrame(rows)[[\"case\", \"recipe\", \"status\", \"error\"]]\n", "coverage" ] }, { "cell_type": "markdown", "id": "anexo-4", "metadata": {}, "source": [ "## Métricas y tiempos completos\n", "\n", "PD creciente de riesgo. AUC, Gini=2AUC−1, KS, Brier, log-loss, PD media, tasa observada y PSI con cortes de Desarrollo. Tiempos en segundos: mediana y rango de tres procesos secuenciales; calentamiento excluido. `fit_seconds` es el pipeline bayesrisk sin el paso de informe; en referentes es su fit. `report_seconds` mide HTML/Word, no Excel ni el verificador del arnés. No son costes funcionalmente equivalentes entre herramientas.\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "anexo-5", "metadata": { "execution": { "iopub.execute_input": "2026-10-19T15:29:43.009291Z", "iopub.status.busy": "2026-10-19T15:29:43.009291Z", "iopub.status.idle": "2026-10-19T15:29:43.039124Z", "shell.execute_reply": "2026-10-19T15:29:43.039124Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
caserecipestatuserrorneventsaucginiksbrierloglossobservedmean_pdpsipsi_binsvariables_finalesvariables_entradafit_seconds_medianfit_seconds_minfit_seconds_maxapply_seconds_medianapply_seconds_minapply_seconds_maxreport_seconds_medianreport_seconds_minreport_seconds_max
0germanfabricaejecutadoNaN250790.7832560.5665110.5284630.1767740.5182920.3160000.2947210.0309791011202.7313132.6828752.9028260.3791100.3752580.3996865.1075365.1036835.491307
1germansin_stepwiseejecutadoNaN250790.7875490.5750980.5303870.1749930.5148270.3160000.2934230.0357621014201.2884121.2611791.3592600.4161520.4091210.4320995.0434465.0044875.368301
2germancinco_binsejecutadoNaN250790.7846250.5692500.5343100.1759580.5159320.3160000.2950190.0277931011202.4083412.3992072.4604930.4639310.3722690.4879545.0281164.9732525.039769
3germanmonotonia_autoejecutadoNaN250790.7894370.5788730.4855280.1742330.5130960.3160000.2980650.0186421010202.4711532.4089392.5576010.3815380.3786990.3969584.9684394.7972235.101882
4germanoptbinning_mipejecutadoNaN250790.8022800.6045600.5488930.1686510.4966910.3160000.2967490.0574071020200.2999090.2863900.3018620.0171450.0166640.0176370.0000000.0000000.000000
5germanscorecardpyejecutadoNaN250790.7728920.5457840.4612480.1822490.5346240.3160000.2970160.0591101020202.9482842.7548623.0696200.1388020.1357650.1492150.0000000.0000000.000000
6germanskorecard_mipejecutadoNaN250790.8030200.6060400.5430450.1682530.4956070.3160000.2972760.0354991020201.5523721.5333681.6361520.1011450.0968050.1046770.0000000.0000000.000000
7taiwanfabricaejecutadoNaN750016750.7665290.5330580.4112280.1390620.4419060.2233330.2182540.00101510102317.77021417.75976218.4577342.3886142.2877232.5128946.6212476.3747606.700636
8taiwansin_stepwiseejecutadoNaN750016750.7675660.5351320.4086680.1389360.4414770.2233330.2182150.00034310142314.02861113.71534216.0029912.9905882.9871023.1516446.9181816.8027617.206478
9taiwancinco_binsejecutadoNaN750016750.7668420.5336840.4127220.1390410.4417820.2233330.2182460.00088310102317.79032917.32873718.0095892.2993752.1912422.3507416.6217986.3554196.626392
10taiwanmonotonia_autoejecutadoNaN750016750.7666190.5332390.4207320.1388710.4414850.2233330.2183120.00233810112318.44243218.27058620.2519122.4427402.4386762.4481166.7846186.7204116.871348
11taiwanoptbinning_mipejecutadoNaN750016750.7716020.5432050.4121610.1380530.4386480.2233330.2181250.0004601023230.9302910.9219460.9719080.3364130.3344960.3544790.0000000.0000000.000000
12taiwanscorecardpyejecutadoNaN750016750.7716080.5432160.4143870.1381860.4388630.2233330.2183970.00175910232312.33450212.00268713.0608544.1416754.0437864.7406340.0000000.0000000.000000
13taiwanskorecard_mipejecutadoNaN750016750.7716180.5432360.4128110.1380490.4386180.2233330.2182270.00054610232320.78568520.59984822.2925326.4960485.6153607.7954040.0000000.0000000.000000
14consumo_comportamientofabricaejecutadoNaN10082390.6560960.3121920.2519060.1712780.5233860.2371030.2333780.00646510561.2952771.2604601.5072130.4416990.4191230.5028334.3860844.3665504.814707
15consumo_comportamientosin_stepwiseejecutadoNaN10082390.6560960.3121920.2519060.1712780.5233860.2371030.2333780.00646510561.0376180.9129871.0608670.4369320.4218200.4488274.3756434.3386944.819603
16consumo_comportamientocinco_binsejecutadoNaN10082390.6536800.3073600.2454040.1716130.5241980.2371030.2328260.00784910561.2680321.1966861.3683110.4240030.4099870.4309734.4217394.2335694.749829
17consumo_comportamientomonotonia_autoejecutadoNaN10082390.6560960.3121920.2519060.1712780.5233860.2371030.2333780.00646510561.2713721.2123321.2898200.5385570.4458670.5513244.3618834.2120584.470849
18consumo_comportamientooptbinning_mipejecutadoNaN10082390.6574260.3148520.2580700.1711680.5229720.2371030.2336520.00741810660.2243300.2125480.2541500.0064240.0061580.0078900.0000000.0000000.000000
19consumo_comportamientoscorecardpyejecutadoNaN10082390.6410080.2820160.2089980.1724920.5280530.2371030.2348340.00596710662.5098682.4703882.5467670.0574260.0543180.0583830.0000000.0000000.000000
20consumo_comportamientoskorecard_mipejecutadoNaN10082390.6573550.3147110.2580700.1711430.5228800.2371030.2336880.00526110660.8279210.7861580.9046690.0741580.0717670.0752010.0000000.0000000.000000
21consumo_driftfabricaejecutadoNaN10455070.7120320.4240630.3196510.2158240.6202050.4851670.4906110.74114010561.3092951.2694031.3160630.4327550.4084280.4368114.4691804.2402934.556495
22consumo_driftsin_stepwiseejecutadoNaN10455070.7120320.4240630.3196510.2158240.6202050.4851670.4906110.74114010560.9999990.9646371.0270160.4313190.4275560.4389244.3830794.3607864.413013
23consumo_driftcinco_binsejecutadoNaN10455070.7097460.4194910.3122240.2163530.6213590.4851670.4906540.74561610561.2845991.2282941.3181330.4064980.4031040.4337444.4783654.2916834.506715
24consumo_driftmonotonia_autoejecutadoNaN10455070.7119890.4239790.3176790.2158440.6203100.4851670.4910730.75214610561.4190661.3585701.8910960.4197460.4057250.4367734.7908834.2944074.794650
25consumo_driftoptbinning_mipejecutadoNaN10455070.7114230.4228460.3127110.2159980.6206120.4851670.4898210.74454210660.2548780.2509080.2591130.0060810.0059910.0062230.0000000.0000000.000000
26consumo_driftscorecardpyejecutadoNaN10455070.6986850.3973700.2877040.2194410.6280870.4851670.4813540.74174910662.0344881.9800452.0868960.0597210.0553880.0668770.0000000.0000000.000000
27consumo_driftskorecard_mipejecutadoNaN10455070.7115070.4230150.3147570.2160020.6206290.4851670.4898340.74305510660.8374140.8341970.8543250.0740320.0725730.0779740.0000000.0000000.000000
\n", "
" ], "text/plain": [ " case recipe status error n events \\\n", "0 german fabrica ejecutado NaN 250 79 \n", "1 german sin_stepwise ejecutado NaN 250 79 \n", "2 german cinco_bins ejecutado NaN 250 79 \n", "3 german monotonia_auto ejecutado NaN 250 79 \n", "4 german optbinning_mip ejecutado NaN 250 79 \n", "5 german scorecardpy ejecutado NaN 250 79 \n", "6 german skorecard_mip ejecutado NaN 250 79 \n", "7 taiwan fabrica ejecutado NaN 7500 1675 \n", "8 taiwan sin_stepwise ejecutado NaN 7500 1675 \n", "9 taiwan cinco_bins ejecutado NaN 7500 1675 \n", "10 taiwan monotonia_auto ejecutado NaN 7500 1675 \n", "11 taiwan optbinning_mip ejecutado NaN 7500 1675 \n", "12 taiwan scorecardpy ejecutado NaN 7500 1675 \n", "13 taiwan skorecard_mip ejecutado NaN 7500 1675 \n", "14 consumo_comportamiento fabrica ejecutado NaN 1008 239 \n", "15 consumo_comportamiento sin_stepwise ejecutado NaN 1008 239 \n", "16 consumo_comportamiento cinco_bins ejecutado NaN 1008 239 \n", "17 consumo_comportamiento monotonia_auto ejecutado NaN 1008 239 \n", "18 consumo_comportamiento optbinning_mip ejecutado NaN 1008 239 \n", "19 consumo_comportamiento scorecardpy ejecutado NaN 1008 239 \n", "20 consumo_comportamiento skorecard_mip ejecutado NaN 1008 239 \n", "21 consumo_drift fabrica ejecutado NaN 1045 507 \n", "22 consumo_drift sin_stepwise ejecutado NaN 1045 507 \n", "23 consumo_drift cinco_bins ejecutado NaN 1045 507 \n", "24 consumo_drift monotonia_auto ejecutado NaN 1045 507 \n", "25 consumo_drift optbinning_mip ejecutado NaN 1045 507 \n", "26 consumo_drift scorecardpy ejecutado NaN 1045 507 \n", "27 consumo_drift skorecard_mip ejecutado NaN 1045 507 \n", "\n", " auc gini ks brier logloss observed mean_pd \\\n", "0 0.783256 0.566511 0.528463 0.176774 0.518292 0.316000 0.294721 \n", "1 0.787549 0.575098 0.530387 0.174993 0.514827 0.316000 0.293423 \n", "2 0.784625 0.569250 0.534310 0.175958 0.515932 0.316000 0.295019 \n", "3 0.789437 0.578873 0.485528 0.174233 0.513096 0.316000 0.298065 \n", "4 0.802280 0.604560 0.548893 0.168651 0.496691 0.316000 0.296749 \n", "5 0.772892 0.545784 0.461248 0.182249 0.534624 0.316000 0.297016 \n", "6 0.803020 0.606040 0.543045 0.168253 0.495607 0.316000 0.297276 \n", "7 0.766529 0.533058 0.411228 0.139062 0.441906 0.223333 0.218254 \n", "8 0.767566 0.535132 0.408668 0.138936 0.441477 0.223333 0.218215 \n", "9 0.766842 0.533684 0.412722 0.139041 0.441782 0.223333 0.218246 \n", "10 0.766619 0.533239 0.420732 0.138871 0.441485 0.223333 0.218312 \n", "11 0.771602 0.543205 0.412161 0.138053 0.438648 0.223333 0.218125 \n", "12 0.771608 0.543216 0.414387 0.138186 0.438863 0.223333 0.218397 \n", "13 0.771618 0.543236 0.412811 0.138049 0.438618 0.223333 0.218227 \n", "14 0.656096 0.312192 0.251906 0.171278 0.523386 0.237103 0.233378 \n", "15 0.656096 0.312192 0.251906 0.171278 0.523386 0.237103 0.233378 \n", "16 0.653680 0.307360 0.245404 0.171613 0.524198 0.237103 0.232826 \n", "17 0.656096 0.312192 0.251906 0.171278 0.523386 0.237103 0.233378 \n", "18 0.657426 0.314852 0.258070 0.171168 0.522972 0.237103 0.233652 \n", "19 0.641008 0.282016 0.208998 0.172492 0.528053 0.237103 0.234834 \n", "20 0.657355 0.314711 0.258070 0.171143 0.522880 0.237103 0.233688 \n", "21 0.712032 0.424063 0.319651 0.215824 0.620205 0.485167 0.490611 \n", "22 0.712032 0.424063 0.319651 0.215824 0.620205 0.485167 0.490611 \n", "23 0.709746 0.419491 0.312224 0.216353 0.621359 0.485167 0.490654 \n", "24 0.711989 0.423979 0.317679 0.215844 0.620310 0.485167 0.491073 \n", "25 0.711423 0.422846 0.312711 0.215998 0.620612 0.485167 0.489821 \n", "26 0.698685 0.397370 0.287704 0.219441 0.628087 0.485167 0.481354 \n", "27 0.711507 0.423015 0.314757 0.216002 0.620629 0.485167 0.489834 \n", "\n", " psi psi_bins variables_finales variables_entrada \\\n", "0 0.030979 10 11 20 \n", "1 0.035762 10 14 20 \n", "2 0.027793 10 11 20 \n", "3 0.018642 10 10 20 \n", "4 0.057407 10 20 20 \n", "5 0.059110 10 20 20 \n", "6 0.035499 10 20 20 \n", "7 0.001015 10 10 23 \n", "8 0.000343 10 14 23 \n", "9 0.000883 10 10 23 \n", "10 0.002338 10 11 23 \n", "11 0.000460 10 23 23 \n", "12 0.001759 10 23 23 \n", "13 0.000546 10 23 23 \n", "14 0.006465 10 5 6 \n", "15 0.006465 10 5 6 \n", "16 0.007849 10 5 6 \n", "17 0.006465 10 5 6 \n", "18 0.007418 10 6 6 \n", "19 0.005967 10 6 6 \n", "20 0.005261 10 6 6 \n", "21 0.741140 10 5 6 \n", "22 0.741140 10 5 6 \n", "23 0.745616 10 5 6 \n", "24 0.752146 10 5 6 \n", "25 0.744542 10 6 6 \n", "26 0.741749 10 6 6 \n", "27 0.743055 10 6 6 \n", "\n", " fit_seconds_median fit_seconds_min fit_seconds_max \\\n", "0 2.731313 2.682875 2.902826 \n", "1 1.288412 1.261179 1.359260 \n", "2 2.408341 2.399207 2.460493 \n", "3 2.471153 2.408939 2.557601 \n", "4 0.299909 0.286390 0.301862 \n", "5 2.948284 2.754862 3.069620 \n", "6 1.552372 1.533368 1.636152 \n", "7 17.770214 17.759762 18.457734 \n", "8 14.028611 13.715342 16.002991 \n", "9 17.790329 17.328737 18.009589 \n", "10 18.442432 18.270586 20.251912 \n", "11 0.930291 0.921946 0.971908 \n", "12 12.334502 12.002687 13.060854 \n", "13 20.785685 20.599848 22.292532 \n", "14 1.295277 1.260460 1.507213 \n", "15 1.037618 0.912987 1.060867 \n", "16 1.268032 1.196686 1.368311 \n", "17 1.271372 1.212332 1.289820 \n", "18 0.224330 0.212548 0.254150 \n", "19 2.509868 2.470388 2.546767 \n", "20 0.827921 0.786158 0.904669 \n", "21 1.309295 1.269403 1.316063 \n", "22 0.999999 0.964637 1.027016 \n", "23 1.284599 1.228294 1.318133 \n", "24 1.419066 1.358570 1.891096 \n", "25 0.254878 0.250908 0.259113 \n", "26 2.034488 1.980045 2.086896 \n", "27 0.837414 0.834197 0.854325 \n", "\n", " apply_seconds_median apply_seconds_min apply_seconds_max \\\n", "0 0.379110 0.375258 0.399686 \n", "1 0.416152 0.409121 0.432099 \n", "2 0.463931 0.372269 0.487954 \n", "3 0.381538 0.378699 0.396958 \n", "4 0.017145 0.016664 0.017637 \n", "5 0.138802 0.135765 0.149215 \n", "6 0.101145 0.096805 0.104677 \n", "7 2.388614 2.287723 2.512894 \n", "8 2.990588 2.987102 3.151644 \n", "9 2.299375 2.191242 2.350741 \n", "10 2.442740 2.438676 2.448116 \n", "11 0.336413 0.334496 0.354479 \n", "12 4.141675 4.043786 4.740634 \n", "13 6.496048 5.615360 7.795404 \n", "14 0.441699 0.419123 0.502833 \n", "15 0.436932 0.421820 0.448827 \n", "16 0.424003 0.409987 0.430973 \n", "17 0.538557 0.445867 0.551324 \n", "18 0.006424 0.006158 0.007890 \n", "19 0.057426 0.054318 0.058383 \n", "20 0.074158 0.071767 0.075201 \n", "21 0.432755 0.408428 0.436811 \n", "22 0.431319 0.427556 0.438924 \n", "23 0.406498 0.403104 0.433744 \n", "24 0.419746 0.405725 0.436773 \n", "25 0.006081 0.005991 0.006223 \n", "26 0.059721 0.055388 0.066877 \n", "27 0.074032 0.072573 0.077974 \n", "\n", " report_seconds_median report_seconds_min report_seconds_max \n", "0 5.107536 5.103683 5.491307 \n", "1 5.043446 5.004487 5.368301 \n", "2 5.028116 4.973252 5.039769 \n", "3 4.968439 4.797223 5.101882 \n", "4 0.000000 0.000000 0.000000 \n", "5 0.000000 0.000000 0.000000 \n", "6 0.000000 0.000000 0.000000 \n", "7 6.621247 6.374760 6.700636 \n", "8 6.918181 6.802761 7.206478 \n", "9 6.621798 6.355419 6.626392 \n", "10 6.784618 6.720411 6.871348 \n", "11 0.000000 0.000000 0.000000 \n", "12 0.000000 0.000000 0.000000 \n", "13 0.000000 0.000000 0.000000 \n", "14 4.386084 4.366550 4.814707 \n", "15 4.375643 4.338694 4.819603 \n", "16 4.421739 4.233569 4.749829 \n", "17 4.361883 4.212058 4.470849 \n", "18 0.000000 0.000000 0.000000 \n", "19 0.000000 0.000000 0.000000 \n", "20 0.000000 0.000000 0.000000 \n", "21 4.469180 4.240293 4.556495 \n", "22 4.383079 4.360786 4.413013 \n", "23 4.478365 4.291683 4.506715 \n", "24 4.790883 4.294407 4.794650 \n", "25 0.000000 0.000000 0.000000 \n", "26 0.000000 0.000000 0.000000 \n", "27 0.000000 0.000000 0.000000 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "metrics = pd.read_csv(\"aggregate/metrics.csv\", float_precision=\"round_trip\")\n", "with pd.option_context(\"display.max_rows\", None, \"display.max_columns\", None):\n", " display(metrics)" ] }, { "cell_type": "markdown", "id": "anexo-6", "metadata": {}, "source": [ "## Incertidumbre pareada\n", "\n", "1.000 remuestreos de grupos completos del test, con reemplazo y semilla 20261016. Diferencia de AUC respecto de fábrica. Intervalo exploratorio condicionado a modelos y partición: no estima variación de entrenamiento ni corrige comparaciones múltiples. Una réplica de una sola clase impide publicar el intervalo; el contador de inválidas se conserva.\n" ] }, { "cell_type": "code", "execution_count": 4, "id": "anexo-7", "metadata": { "execution": { "iopub.execute_input": "2026-10-19T15:29:43.039124Z", "iopub.status.busy": "2026-10-19T15:29:43.039124Z", "iopub.status.idle": "2026-10-19T15:29:43.059492Z", "shell.execute_reply": "2026-10-19T15:29:43.059492Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
caserecipereplicatesinvalidgroupsseeddelta_aucinterval_95limitation
0germansin_stepwise10000250202610160.004293[-0.0040868826912320005, 0.013577331892894002]Condicionado a modelos y partición; sin varian...
1germancinco_bins10000250202610160.001369[-0.0029641620015810003, 0.0055617679961360005]Condicionado a modelos y partición; sin varian...
2germanmonotonia_auto10000250202610160.006181[-0.012938887548262001, 0.027026433949529003]Condicionado a modelos y partición; sin varian...
3germanoptbinning_mip10000250202610160.019024[0.0018707663264580002, 0.036382840173672]Condicionado a modelos y partición; sin varian...
4germanscorecardpy1000025020261016-0.010363[-0.034800563922153, 0.015076086241399]Condicionado a modelos y partición; sin varian...
5germanskorecard_mip10000250202610160.019765[0.0010018690543070002, 0.037743267588342004]Condicionado a modelos y partición; sin varian...
6taiwansin_stepwise100007486202610160.001037[-1.9509952827898022e-05, 0.001966785230722]Condicionado a modelos y partición; sin varian...
7taiwancinco_bins100007486202610160.000313[-0.000200181714715, 0.000824817829918]Condicionado a modelos y partición; sin varian...
8taiwanmonotonia_auto100007486202610160.000091[-0.001977883129725, 0.002387708450899]Condicionado a modelos y partición; sin varian...
9taiwanoptbinning_mip100007486202610160.005074[0.001357412372063, 0.008847490093935]Condicionado a modelos y partición; sin varian...
10taiwanscorecardpy100007486202610160.005079[0.000365997770776, 0.009941582646820001]Condicionado a modelos y partición; sin varian...
11taiwanskorecard_mip100007486202610160.005089[0.0012437343840110001, 0.008907379850505001]Condicionado a modelos y partición; sin varian...
12consumo_comportamientosin_stepwise100001008202610160.000000[0.0, 0.0]Condicionado a modelos y partición; sin varian...
13consumo_comportamientocinco_bins10000100820261016-0.002416[-0.006137073196538, 0.0010826962778820001]Condicionado a modelos y partición; sin varian...
14consumo_comportamientomonotonia_auto100001008202610160.000000[0.0, 0.0]Condicionado a modelos y partición; sin varian...
15consumo_comportamientooptbinning_mip100001008202610160.001330[-0.000941415060655, 0.0038230911349280004]Condicionado a modelos y partición; sin varian...
16consumo_comportamientoscorecardpy10000100820261016-0.015088[-0.029414331058504002, -2.1432452071573763e-05]Condicionado a modelos y partición; sin varian...
17consumo_comportamientoskorecard_mip100001008202610160.001260[-0.001202462691238, 0.0037542941130990003]Condicionado a modelos y partición; sin varian...
18consumo_driftsin_stepwise100001045202610160.000000[0.0, 0.0]Condicionado a modelos y partición; sin varian...
19consumo_driftcinco_bins10000104520261016-0.002286[-0.006226048118074001, 0.0014337465264500002]Condicionado a modelos y partición; sin varian...
20consumo_driftmonotonia_auto10000104520261016-0.000042[-0.0007069310491020001, 0.000640194024281]Condicionado a modelos y partición; sin varian...
21consumo_driftoptbinning_mip10000104520261016-0.000609[-0.0023342094297450003, 0.0010808388196210002]Condicionado a modelos y partición; sin varian...
22consumo_driftscorecardpy10000104520261016-0.013347[-0.024439642352164, -0.0025977202483750002]Condicionado a modelos y partición; sin varian...
23consumo_driftskorecard_mip10000104520261016-0.000524[-0.0022562920617530003, 0.0011131261142110001]Condicionado a modelos y partición; sin varian...
\n", "
" ], "text/plain": [ " case recipe replicates invalid groups \\\n", "0 german sin_stepwise 1000 0 250 \n", "1 german cinco_bins 1000 0 250 \n", "2 german monotonia_auto 1000 0 250 \n", "3 german optbinning_mip 1000 0 250 \n", "4 german scorecardpy 1000 0 250 \n", "5 german skorecard_mip 1000 0 250 \n", "6 taiwan sin_stepwise 1000 0 7486 \n", "7 taiwan cinco_bins 1000 0 7486 \n", "8 taiwan monotonia_auto 1000 0 7486 \n", "9 taiwan optbinning_mip 1000 0 7486 \n", "10 taiwan scorecardpy 1000 0 7486 \n", "11 taiwan skorecard_mip 1000 0 7486 \n", "12 consumo_comportamiento sin_stepwise 1000 0 1008 \n", "13 consumo_comportamiento cinco_bins 1000 0 1008 \n", "14 consumo_comportamiento monotonia_auto 1000 0 1008 \n", "15 consumo_comportamiento optbinning_mip 1000 0 1008 \n", "16 consumo_comportamiento scorecardpy 1000 0 1008 \n", "17 consumo_comportamiento skorecard_mip 1000 0 1008 \n", "18 consumo_drift sin_stepwise 1000 0 1045 \n", "19 consumo_drift cinco_bins 1000 0 1045 \n", "20 consumo_drift monotonia_auto 1000 0 1045 \n", "21 consumo_drift optbinning_mip 1000 0 1045 \n", "22 consumo_drift scorecardpy 1000 0 1045 \n", "23 consumo_drift skorecard_mip 1000 0 1045 \n", "\n", " seed delta_auc interval_95 \\\n", "0 20261016 0.004293 [-0.0040868826912320005, 0.013577331892894002] \n", "1 20261016 0.001369 [-0.0029641620015810003, 0.0055617679961360005] \n", "2 20261016 0.006181 [-0.012938887548262001, 0.027026433949529003] \n", "3 20261016 0.019024 [0.0018707663264580002, 0.036382840173672] \n", "4 20261016 -0.010363 [-0.034800563922153, 0.015076086241399] \n", "5 20261016 0.019765 [0.0010018690543070002, 0.037743267588342004] \n", "6 20261016 0.001037 [-1.9509952827898022e-05, 0.001966785230722] \n", "7 20261016 0.000313 [-0.000200181714715, 0.000824817829918] \n", "8 20261016 0.000091 [-0.001977883129725, 0.002387708450899] \n", "9 20261016 0.005074 [0.001357412372063, 0.008847490093935] \n", "10 20261016 0.005079 [0.000365997770776, 0.009941582646820001] \n", "11 20261016 0.005089 [0.0012437343840110001, 0.008907379850505001] \n", "12 20261016 0.000000 [0.0, 0.0] \n", "13 20261016 -0.002416 [-0.006137073196538, 0.0010826962778820001] \n", "14 20261016 0.000000 [0.0, 0.0] \n", "15 20261016 0.001330 [-0.000941415060655, 0.0038230911349280004] \n", "16 20261016 -0.015088 [-0.029414331058504002, -2.1432452071573763e-05] \n", "17 20261016 0.001260 [-0.001202462691238, 0.0037542941130990003] \n", "18 20261016 0.000000 [0.0, 0.0] \n", "19 20261016 -0.002286 [-0.006226048118074001, 0.0014337465264500002] \n", "20 20261016 -0.000042 [-0.0007069310491020001, 0.000640194024281] \n", "21 20261016 -0.000609 [-0.0023342094297450003, 0.0010808388196210002] \n", "22 20261016 -0.013347 [-0.024439642352164, -0.0025977202483750002] \n", "23 20261016 -0.000524 [-0.0022562920617530003, 0.0011131261142110001] \n", "\n", " limitation \n", "0 Condicionado a modelos y partición; sin varian... \n", "1 Condicionado a modelos y partición; sin varian... \n", "2 Condicionado a modelos y partición; sin varian... \n", "3 Condicionado a modelos y partición; sin varian... \n", "4 Condicionado a modelos y partición; sin varian... \n", "5 Condicionado a modelos y partición; sin varian... \n", "6 Condicionado a modelos y partición; sin varian... \n", "7 Condicionado a modelos y partición; sin varian... \n", "8 Condicionado a modelos y partición; sin varian... \n", "9 Condicionado a modelos y partición; sin varian... \n", "10 Condicionado a modelos y partición; sin varian... \n", "11 Condicionado a modelos y partición; sin varian... \n", "12 Condicionado a modelos y partición; sin varian... \n", "13 Condicionado a modelos y partición; sin varian... \n", "14 Condicionado a modelos y partición; sin varian... \n", "15 Condicionado a modelos y partición; sin varian... \n", "16 Condicionado a modelos y partición; sin varian... \n", "17 Condicionado a modelos y partición; sin varian... \n", "18 Condicionado a modelos y partición; sin varian... \n", "19 Condicionado a modelos y partición; sin varian... \n", "20 Condicionado a modelos y partición; sin varian... \n", "21 Condicionado a modelos y partición; sin varian... \n", "22 Condicionado a modelos y partición; sin varian... \n", "23 Condicionado a modelos y partición; sin varian... " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "intervals = pd.read_json(\"aggregate/bootstrap.json\")\n", "intervals" ] }, { "cell_type": "markdown", "id": "anexo-8", "metadata": {}, "source": [ "## Artefactos, configuración y reproducción\n", "\n", "`artifacts/` contiene HTML, Word y Excel de las 16 corridas bayesrisk. `results/tables/` conserva las tablas de los tres referentes y los resúmenes. `expected/` contiene el contenido esperado capturado de los Studies propios, sin distribuir pickles. `prepared/` conserva fuentes transformadas, índices, grupos, muestras y manifiestos; `cache/` contiene sólo las dos fuentes UCI autorizadas. El mapa `provenance.json` distingue bytes originales y derivados con rutas portables.\n", "\n", "Las cifras de este cuaderno pertenecen a la evaluación exterior. No se presentan como métricas del Holdout interior ni se construyen Studies de terceros para Comparison.\n", "\n", "Para reproducir después de instalar el entorno y el wheel candidato exacto descritos en la guía, inicia un proceso con `PYTHONHASHSEED=0` y ejecuta:\n", "\n", "```text\n", "python reproduce.py\n", "```\n", "\n", "El script crea `reproduction/`, rechaza sobrescribir una reproducción previa, comprueba fuentes y conserva las 28 filas incluso ante fallos. La falta de red con cache ausente o una fuente alterada bloquea la ejecución; no se interpreta como éxito. No exige credenciales.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (H10)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.10" } }, "nbformat": 4, "nbformat_minor": 5 }