o b@@srddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z GdddZdS)N)Plumi)Vemli)Runer)Sidatc@sveZdZdddZdddZddZdd d Zd d Zd dZdddZ dddZ ddZ ddZ ddZ ddZdS)TrainFcCst||_t|j|_t|j|_t|j|_t|j|_|jj |_ d|_ d|_ d|jj vr5t |jj d|_ ||_dS)NFtrain_limit_by_days)erplumirvemlirrunerrsidatsig has_error train_limitconfint do_verbose)selferimprr!/home/ernad/ernad/python/train.py__init__s      zTrain.__init__cCs|j}||jvrtd|tjddS|j|}||_||_|s#i|_|j}|jdd|d|_ |jdd|d|_ |jdd|d|_ |jdd|d}t |dS) Nztrain does not see the repcode fileFmocla/.train.modelz.sutra.json.gz)rreportprintsysstderrrepcodesutrar dirs train_fufi model_fufi sutra_fufifilerprepare)rr" grund_boostrrr r%rrrsetup s"  z Train.setupcCs|j}|||j}|j|jd}|j|}|durdS|jd}|jj|dd}|dur2dStj |dkr build the train file, only done if there is no train filezmocla: start buildtrain deletes the empty train deletes z based on Fwr+1-1NTztrain skips plumis older than ztrain skips plumis from is_seedabletrain: z is too old for seeds 0000-00-00z%train does not have sidat vemlis, no seednote+1  z(train does not see external seed krikts grund_sizezI add z from grund.rz/grund*).rrr+r%r&r# seen_papidsr(r)r2r3isfileis_emptyremoveopen train_filer fufisam_i_older_than_train_limit issuedaterdaterago plumi_fufi feed_issuerrr" finish_buildr sidis_fufi add_sidis krikt_fufir build_fufiloadcheck_seen_papidswrite feed_grundrlenstrfind_grund_to_addrr$glob)rr" grund_papidsrr%r& plumi_dafushave_i_printed_about_skiprQrWkrikt_seed_fufiseed_vemli_fufi vemli_datapapid grund_gap grund_globfufirrr build_fileHs                              zTrain.build_filecCsft|jdkrtdtjddS|jrtd|jt|j|j|jr,td|j |j dS)Nrz#train does not write an empty sutrarFz train writes ztrain finishes building T) r_r#rr r!rr'r(dumpr%rNclose)rrrrrVs zTrain.finish_buildc Cstj|sddl}|jd|jddt|}|j}|D]_}|j |||}||}|durat d|j |j|j |||}|durad|d|}|d |f7}t |t jd q|j||d |jd |d |jddd7<qdS)zThis adds the interla seedrNzsidat T)shellz+train got a None sidis vemli, try to updatez train for z no z for sidis rrCrErFrAr=rG)r2r3rJ subprocessrunr"r(r[r greprr update_reportr r!r\rNr]r#) rrWrqpapisur"rir rQerrorrrrrXs*  zTrain.add_sidiscCs||jvr9|durtd|dtjdd|_dS|dkr'td|ddStd|d|tjdd|_dS|j|dS) N train sees zagain.rTFgrund in the grund.z in )rIrr r!rappend)rrirDrrrr\s  zTrain.check_seen_papidscCs&|j||d|j}tj|r't|r'td|dt|| |t |jdtj|sQ|j r=td|| |tj|sQtd|t j d|j rZtd|t|jrrtd |jt j dt|jd S|j}t||gstd |d |dd S|jrtdtd|t|td}d|jjvr||jjdd7}||d7}||7}d|d|j}|j rt|d||jj||dd}|d urtd|dd S|j rtdtj|jddr td|jdt j dt|jd Stj |dddd S) z;--> run training, build train file only if no training filer*r:.rz train builds ztrain could not build the rz train uses ztrain has no data in Nztrain doneres z over Fzmocla: end because of errorzmocla: I delete z/usr/bin/svm-train -b 1 svm_train_flags train__T)do_skipr@z is lockedzsvm-train doneX)min_sizeztrain finds that the model z is empty. It deletes it.r)r)r+r%r2r3rJr(rKrrLrmclear_by_extensionrr r!r&r5rquitrrr r run_with_lock)rr"r*r%r&runnerloggeroutrrrrrsn                z Train.runcCs|j}|jr td|d|jd}|jdd||}tj|sHtd|dtj dtd |j d t |j td tj dt t |jd }|}t||_d |_|D]}|j|dd|d||q^|j|jdS)zfeeds issue into train fileztrain: build_file adds rz.json.gzr rztrain does not see z' this should have done by post_process.rr;r|z train ends.rrCN)rrrr r$r2r3rJr r!r%rLrgziprMrT readlinesr(r[vstringr\decodefeed_plumi_linerstriprNr])rrQrend vemli_fufi plumi_file plumi_lineslinerrrrU)s.    zTrain.feed_issuecCs*ddlm}||j|_|jj}|dur$tdtjdt|j t |j j j |vr8td|j j j d|t|jj}|D]}||vrTtd|d d tjdqAt|j d |_|D]4}||jvrntd |d q^|j|ddsvq^||vr{q^|jd||d|jddd7<q^dS)z'--> add only the papids in grund_papidsr)GrundNz9train can't continue without grund vemlis, try 'grund -l'rzsig inconsistency between z and ztrain: the requested z is not in the grund,z try to run grund -laztrain finds seen paper ryrxrCz-1 rFrAr>rGT)rxrrrrr r!r2rLr%rr r r Exceptionr(r[rMrNrIr\r]r#)rrcrgrund_vemli_fufi grund_vemlisrirrrr^AsL     zTrain.feed_grundcCs||j}|dd}|dd}||vr!t|d|jtjddS|j|j|d7<|j|d||d7_dS) Nrrz is not in the vemli of rrGr~rF)rrrQr r!r#r)rrrindicrirrrrjs  "zTrain.feed_plumi_linec Cs|jtd|j|jddddlm}||jd|_td|jj|jdd|jd }td |dd l m }||jd|_ |j j |d }d}t |jj d |}g}|D]} ||krc|d7}qX|| qXt||krtdtt|dt|tjd|S)Nztrain runs for grundTr{r)Mocla)rztrain runs grund classF)do_asembrrw)Asemb)rlrHrGzcount problem r~r)rNrorrrr"rrrrasembrparse_side_fufirrrzr_r`r r!) rrjr side_fufirpapwescountstartrcrirrrraxs8      zTrain.find_grund_to_add)F)N)__name__ __module__ __qualname__rr+r5rmrVrXr\rrrUr^rrarrrrrs   s  6) r)rbrr2r r(rRr rr rr rr rrrrrrs