a b@@srddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z GdddZdS)N)Plumi)Vemli)Runer)Sidatc@sveZdZdddZdddZddZdd d Zd d Zd dZdddZ dddZ ddZ ddZ ddZ ddZdS) TrainFcCst||_t|j|_t|j|_t|j|_t|j|_|jj |_ d|_ d|_ d|jj vrjt |jj d|_ ||_dS)NFZtrain_limit_by_days)erplumirvemlirrunerrsidatsig has_error train_limitconfint do_verbose)selferimprr!/home/ernad/ernad/python/train.py__init__s      zTrain.__init__cCs|j}||jvr&td|tjddS|j|}||_||_|sFi|_|j}|jdd|d|_ |jdd|d|_ |jdd|d|_ |jdd|d}t |dS) Nztrain does not see the repcode fileFmocla/.train.modelz.sutra.json.gz)rreportprintsysstderrrepcodesutrar dirs train_fufi model_fufi sutra_fufifilerprepare)rr! grund_boostrrr r$rrrsetup s"  z Train.setupcCs|j}|||j}|j|jd}|j|}|dur>dS|jd}|jj|dd}|durddStj |dkrxdSt ||gS) z4no renew if *any* model is newer than last sent filesentNFrz*.model)searchTd) rr*rr#d very_last last_by_mtimeospathgetsizer'donere)rr!rrsent_dirZlast_sent_file mocla_dir last_modelrrrr45s   z Train.donereNcCs |j}|rtd|||j}|j}i|_g|_t|t j |rt |rtd|t |t j |jrtd|d|t |ndSt|d|_|j|}d}d|_|D]}||_i|j|<d|j|d<d|j|d <|jd ur\t||jkr\d |_d |j|d<d |j|d <|sHtd |d }|jrtd |q|||_||qd|jjvs|jrtd|jd|d Si|jd<d|jdd<d|jdd <|j|} || |j |} t j | rt|j!"| } | d ur td| nRt#| } | D]B} |j$| dd|j%d| | d|jddd7<q.n td| |d ur|&|t'|jjdt(|j}|dkrtdt)|d|*|}|&|n2|jj+|j,dd}t--|D]}t |q|d S)z=--> build the train file, only done if there is no train filezmocla: start buildtrain deletes the empty train deletes z based on Fwr+1-1NTztrain skips plumis older than ztrain skips plumis from Z is_seedabletrain: z is too old for seeds 0000-00-00z%train does not have sidat vemlis, no seednote+1  z(train does not see external seed krikts grund_sizezI add z from grund.rz/grund*).rrr*r$r%r" seen_papidsr'r(r1r2isfileis_emptyremoveopen train_filerfufisZam_i_older_than_train_limit issuedaterdaterago plumi_fufi feed_issuerrr! finish_buildr sidis_fufi add_sidisZ krikt_fufir Z build_fufiloadcheck_seen_papidswrite feed_grundrlenstrfind_grund_to_addrr#glob)rr! grund_papidsrr$r%Z plumi_dafusZhave_i_printed_about_skiprMrSZkrikt_seed_fufiZseed_vemli_fufiZ vemli_datapapid grund_gapZ grund_globfufirrr build_fileHs                             zTrain.build_filecCsft|jdkr tdtjddS|jr4td|jt|j|j|jrXtd|j |j dS)Nrz#train does not write an empty sutrarFz train writes ztrain finishes building ) rYr"rrr rr&r'dumpr$rKclose)rrrrrRszTrain.finish_buildc Cstj|s(ddl}|jd|jddt|}|j}|D]}|j |||}||}|durt d|j |j|j |||}|durd|d|}|d |f7}t |t jd q<|j||d |jd |d |jddd7<qr;rD)r1r2rG subprocessrunr!r'rUr Zgreprr Z update_reportrr rVrKrWr") rrSreZpapisur!r^r rMerrorrrrrTs*  zTrain.add_sidiscCs||jvrr|dur2td|dtjdd|_dS|dkrNtd|ddStd|d|tjdd|_dS|j|dS) N train sees zagain.rTFgrund in the grund.z in )rFrrr r append)rr^rArrrrVs  zTrain.check_seen_papidscCs2|j||d|j}tj|rNt|rNtd|dt|| |t |jdtj|s|j rztd|| |tj|std|t j d|j rtd|t|jrtd |jt j dt|jd S|j}t||gstd |d |dd S|jr@tdtd|t|td}d|jjvrr||jjdd7}||d7}||7}d|d|j}|j rt|d||jj||dd}|d urtd|dd S|j rtdtj|jddrtd|jdt j dt|jd Stj |dddd S) z;--> run training, build train file only if no training filer)r8.rz train builds ztrain could not build the rz train uses ztrain has no data in Nztrain doneres z over Fzmocla: end because of errorzmocla: I delete z/usr/bin/svm-train -b 1 Zsvm_train_flags Ztrain__T)Zdo_skipr=z is lockedzsvm-train doneX)min_sizeztrain finds that the model z is empty. It deletes it.r)r)r*r$r1r2rGr'rHrrIraclear_by_extensionrrr r%r4r quitrrr r Z run_with_lock)rr!r)r$r%Zrunnerloggeroutrrrrfsn                z Train.runcCs|j}|jrtd|d|jd}|jdd||}tj|std|dtj dtd |j d t |j td tj dt t |jd }|}t||_d |_|D]0}|j|dd|d||q|j|jdS)zfeeds issue into train fileztrain: build_file adds roz.json.gzr rztrain does not see z' this should have done by post_process.rr9rmz train ends.rr@N)rrrr r#r1r2rGrr r$rIrsgziprJrP readlinesr'rUvstringrVdecodefeed_plumi_linerstriprKrW)rrMrend vemli_fufiZ plumi_fileZ plumi_lineslinerrrrQ(s.    zTrain.feed_issuecCs*ddlm}||j|_|jj}|durHtdtjdt|j t |j j j |vrptd|j j j d|t|jj}|D]&}||vrtd|d d tjdqt|j d |_|D]h}||jvrtd |d q|j|ddsq||vrq|jd||d|jddd7<qdS)z'--> add only the papids in grund_papidsr)GrundNz9train can't continue without grund vemlis, try 'grund -l'rzsig inconsistency between z and ztrain: the requested z is not in the grund,z try to run grund -laztrain finds seen paper rjrir@z-1 rCr>r<rDT)rirrrrrr r1rIr$rsr r r Exceptionr'rUrJrKrFrVrWr")rr]rZgrund_vemli_fufiZ grund_vemlisr^rrrrX@sJ     zTrain.feed_grundcCs||j}|dd}|dd}||vrBt|d|jtjddS|j|j|d7<|j|d||d7_dS) Nrrxz is not in the vemli of rrDrnrC)r|rrMrr r"r})rrr|Zindicr^rrrris  zTrain.feed_plumi_linec Cs |jtd|j|jddddlm}||jd|_td|jj|jdd|jd }td |dd l m }||jd|_ |j j |d }d}t |jj d |}g}|D] } ||kr|d7}q|| qt||krtdtt|dt|tjd|S)Nztrain runs for grundTrlr)Mocla)rztrain runs grund classF)Zdo_asembrrh)Asemb)r`rErDzcount problem rnr)rKrcrrfr!rrrrZasembrZparse_side_fufirrrkrYrZrr ) rr_rZ side_fufirZpapwescountstartr]r^rrrr[ws8     zTrain.find_grund_to_add)F)F)N)N)F)__name__ __module__ __qualname__rr*r4rarRrTrVrfrQrXrr[rrrrrs   s   6)r)r\rzr1rr'rNrrr rr rr rrrrrrs