Add a simpler Random Optimizer #173
Labels
Comments
wynot12
pushed a commit
that referenced
this issue
Nov 2, 2017
yunseong
pushed a commit
that referenced
this issue
Nov 3, 2017
* Initial commit * [ET-1] Required initial setup (#2) * pom.xml * Dependency settings including REEF, evaluator-manager, and Avro * Plugin settings including shade, Rat, and checkstyle * checkstyle.xml for style consistency rules * .gitignore with some more rules for the project * package-info.java for package edu.snu.cay.services.et * [MINOR] pom.xml: remove dependencyManagement, pluginManagement (#4) * pom.xml: remove dependencyManagement, pluginManagement * Fix * Remove use of avro plugins which is not used yet * [ET-3] Container abstraction with network connectivity (#6) * Container abstraction: Jangho's version * Rename ExactContainerManager to ContainerManagerImpl * Refine * Remove table-related components * Refine * Minor updates * Reuse connection * Remove interface of ContainerManager * Add test running an example app * Refine comments * Update example app name in launch script * Refine comments * Resolve YS's comments * Fix style error * Change ResouceConfiguration not to use REEF configuration * Rename ETDriver to ETMaster * Minor refine * [MINOR] Change checkstyle rule to allow no Javadoc in NamedParameters (#18) * [ET-7] Implement driver-side Table abstraction (#9) * Add table-related parameters and configuration * Add table-related components * Update existing components to work with table * Fix test to catch failure * Minor update * Fix comments in named parameters * Change checkstyle to allow missing Javadoc for NamedParameters like Cay * Remove unnecessary code * Add Update and Partition functions * Minor refine * Add Table interface * Remove table subscription part * Remove unnecessary part * Restore example app to use only one container group * Resolve YS's comments * Initialization implies initial association and allocation of table * Refine FilePath named parameter * Refine comments * Fix a style error * Update comments * Minor * Update table init part * Refine comments * Resolve YS's comments * Minor updates * Change TableMetadata to CreatedTable * Table init: Created, Associated, Allocated * Refine comments * Update comments * Refine comments * Rework about table init flow: RawTable -> MaterializedTable * Make comments and naming consistent with minor rewrite * Minor updates * [MINOR] Rename Container as Executor (#20) * [MINOR] Rename components according to our new design (#21) * [MINOR] Rename components according to our new design * Container => Executor * materialize => allocate * Partition => Tablet * [ET-8] Implement avro protocol and msg sender interfaces for master and executor (#24) * [CAY-10, 28] Implement message sender/handler and table initializer (#29) * [ET-11, 22] Evaluator-side Table abstraction (#30) * TableAccessor interface * [MINOR] Remove RawTable and use injection for instantiating tables in master (#32) * Remove raw table and use injection for instantiating tables in driver * Resolve comments * Minor refine * Fix nits * Minor comment * DataAccess Avro minor fix (#35) * [ET-12,33] Implement OwnershipCache and TabletManager (#34) * Remove raw table and use injection for instantiating tables in driver * Resolve comments * Minor refine * Implement TabletManager and OwnershipCache * Remove a notion of tablet * Update copyright years * [CAY-25] Implement data access routine including local and remote access (#36) * Implement data access routine inclusing local and remote access * Update comments and refine code * Separate state (mutable) and metadata (immutable) of DataOpertion * Resolve comments * Minor refine * Minor * OrigId always exists * Minor update * [MINOR] Change the type of data_op_id to long * [ET-39] Make submitTask return a future of TaskResult (#40) * SubmitTask returns a future of TaskResult * Refactoring SimpleET tasks * [MINOR] Separate remote op sender and handler * [ET-13] Implement data migration (#41) * Minor * minor updates * Fix concurrency issue which might happen in local runtime * [MINOR] Instantiate RemoteOpHandler/Sender at Tables (#45) * Instantiate RemoteOpHandler/Sender at Tables * Fix loopy constructors * Merge master * [MINOR] Add missing executor parameter (#49) * [ET-47] Add unit tests for remote access op sender and handler (#48) * Add tests for remote acces op sender and handler * Add comments * Minor fix * [ET-26] Support bulk data loading from a file to a table (#51) * Support bulk data loading from file to table * Minor fix * Fix * Resolve YS's comments * Minor update * Minor update * [ET-54] Allow user configuration within executor (#55) * seperate context and service configuration * Allow user configuration with executor * [MINOR] Throw RuntimeException in `SimpleET`, instead of assertion * [ET-56] Implement AddInteger example app (#57) * AddInteger app * Fix a style error * Add this app to integration test * Pass task parameters using task configuration * Update comments * Refine comments * Update argument requirement * Delete IntegerCodec and use SerializableCodec * Exclude driver eval count for the max number of evals in local runtime * [ET-37] Lower the logging level of NetworkLinkListener * [ET-27] Provide table APIs for accessing local data (#61) * Provide an iterator on local blocks * API for obtaining local data at once * Remove unnecessary syncronized * Refactoring removeBlock method * Fix a style error * Resolve YS's comments * Change Map.Entry to Entry in interface * Minor updates * [ET-53] Introduce data parser (#63) * Introduce DataParser * Use DataParser in SimpleET * Fix comments * [ET-64] Allow user param for table configuration (#65) * Allow user params in TableConfiguration * Change AddInteger app to use user parameters for table * Change TableManager to use empty injector for instantiating table * update copyright * Fix a style error * [MINOR] Return builder when setting data parser to TableConfigurationBuilder * [ET-67] Support update value type that is different from value type (#70) * commit * Minor revision * [ET-72] Allow users to configure custom services (#73) * allow user services * Add example app using aggregation service * update copyright years * minor refine * Fix potential NPE * Renamed classes * Minor refinement * [ET-58] Implement async APIs for accessing tables (#71) * commit * Add comments * inverse async boolean * Update comments * Minor fix * [ET-75] Support PutIfAbsent API (#76) * Add putIfAbsent API * test code * Rename PUTIFABSENT to PUT_IF_ABSENT for readability * [MINOR] Fix a bug in HashBasedBlockPartitioner to get hash value correctly #77 * [MINOR] Encapsulate configurations for adding an executor #79 * [MINOR] Introduce Tablet, which is an abstraction of local portion of Table (#80) * Introduce tablet * Separate out TableComponents from TableImpl * Remove tableComponent interface * Minor revision in comment * [ET-82] Support NoReply methods for writing to tables (#83) * commit * Resolve YS's comments * Minor updates on comments * Fix a bug * Change style of timeout check * [ET-85] Implement Table drop routine (#86) * commit * Rename: delete -> drop, TableInitializer -> TableControlAgent * Delete -> Drop * Minor fix * Fix checkstyle violation * minor * remove block line * Minor rename * [MINOR] Apply name change of Aggregation service to CentComm service * [ET-91] Async non-blocking version of ET operations (#94) * Add ListenableFuture and its implementations * Change ET's operation methods to return ListenableFuture * Fix style error * Update copyright years. nit * Add comments. Minor refine * minor refine * Reorg package structure * Specify return type in Javadoc * Use wildcard when type has no meaning * Fix style error * Fix nits * [ET-89] Collect metrics in ET (#95) * Define message type for metrics * Expose an API to get the numBlocks * Configuration for metric collection service * Apply metric collection in AddInteger * Track hostname as a part of metrics * [MINOR] Modify MetricReceiver that informs the source id (#101) * [MINOR] Modify metric receiver that informs the source id * [MINOR] Change CustomMetricsCodec to the ordinary Codec (#104) * Modify CustomMetricsCodec to an ordinary codec * Remove EmptyMetricCodec * [ET-69, 103] Implement PlanExecutor for ET operations (#102) * commit * Add CreateOp and DropOp. Update comments * fix style error * Change ETPlan as a concrete implementation class * Move Op execution code into Op::execute() * Fix style error * Fix comment * Revise comments and remove unused methods * [ET-105,108] Collect minimal metrics for remote ops (#109) * [MINOR] Modify metric receiver that informs the source id * Enable metrics related to network usage * Restrict the metrics to collect: GET * Make it thread-safe * Reduce possible lose of increments * [MINOR] Restore get[Src|Dst]ExecutorId in MoveOp (#110) * [MINOR] Expose the information of mapping between table and partitions * [ET-92] Implement task stop routine (#111) * Implement task stop routine * Make class final and override hashCode() and equals() * minor fix * Minor fix * Minor fix * [MINOR] Fix PlanExecutor to return immediately when plan is empty * [MINOR] Use virtual executor id for AllocateOp with the plan * [ET-93] Implement table unsubscription routine * [MINOR] Requires explicit (un)association between table and executor (#117) * Explicit association and unassociation. * Minor comments * [ET-81] Table access test (#90) * implement CRUD in first case * subscriber single thread test in hash, ordered table * Print log in PlanExecutor (#122) * [MINOR] Fix a bug in incrementing remote access stat (#126) * [MINOR] Fix a bug in incrementing remote access stat * [ET-88] Remove a table entry from TableManager when dropping the table (#119) * [ET-88] remove allocated table in allocatedTableMap * [ET-127] Delegate communication part from computation threads to network threads (#129) * Delegate communication part from computation threads to network threads * Fix a style error * Resolve comments * Minor fix on comments and the use of nullable * refine comment * [MINOR] Make TableControlAgent thread-safe #134 * [MINOR] Fix duplicate registration of remote op in OpSender #132 * [ET-124] Redirect remote access op (#138) * commit * minor * Minor improvement on comments * [ET-125] Resend remote access op when NetworkException while sending msgs (#139) * commit * minor * commit * Fix style errors * minor * fix * Resolve YS's comments * [ET-142] Expose a listener interface for the op completion within plan execution (#143) * [ET-100,113] Make a separate configuration for metric collection service. (#140) * Separate configuration for metric service * Add example * Add named parameters * [ET-141] Introduce getOrInit() API for initializing empty values (#145) * [ET-141] Add getOrInit() method * Change Simple ET to use getOrInit method * [MINOR] Fix op sender's method to be usable for RemoteOpHandler #147 * [MINOR] Update OpHandler to use thread-local queues #148 * [ET-118] Flush out sender/handler queues before removing table from executor (#149) * commit * minor * refine comment * Fix a style error * fix * update * [ET-151] Synchronize ownership when unassociting executors (#152) * commit * minor * Fix a style error * minor * remove executor to be unassociated from target list of sync * resolve YS's comments * minimize synchronized block * refine comment * [MINOR] Fix Optype problem in UnassociateOp (#155) * [MINOR] Fix Optype problem in UnassociateOp (#154) * [ET-96] Make executor deallocation event listenable (#156) * commit * Fix a style error * commit (#154) * [ET-146] Change example apps not to hold REEF's thread in Driver's StartHandler (#157) [ET-146] Change example apps not to hold REEF's thread in Driver's StartHandler * [ET-128] Make remote access parameters configurable (#158) * Move remote access parameters to a separate package * Introduce RemoteAccessConfiguration * Customize remote access configuration in TableAccessET * Add logs in Sender/Handler to check the customized configuration * minor * [MINOR] Changes ETPlan methods public #160 * [MINOR] Fix a concurrency bug in migration #164 * [ET-161] Migration protocol for immutable tables (#163) * temp * fix * Change mutable table setting to be mandatory * update comments * nit * refine * Add logs in migration process * refine comments * remove unused method * minor updates * [MINOR] Optimize bulk data loading (#165) * commit * Remove changes written for debugging * Undo unnecesary changes * Minor comment * [MINOR] fix sporadic failure by reducing the number of executors (#166) * [MINOR] fix sporadic failure by changing the number of executors * [ET-107] Implements integration tests for PlanExecutor (#159) * [MINOR] Log detailed info of plan execution (#167) * Log elapsed time of ops and plan in execution * Log detailed info (e.g., numSentBytes) of migration * [MINOR] Implement a plan executor that prints log messages only (#169) * [ET-170] Implement a fallback logic for remote data access (#172) * [ET-168] Dynamic metric collection service (#171) * commit * Change example apps to use new metric service * minor * minor * minor * Make MetricET app change metric policy dynamically * clean accumulated metrics before starting with a new policy * minor * Resolve YS's comments * Minor update in log message * [MINOR] Introduce a method for acquiring the number of data in local Tablets (#173) * [MINOR] Start/stop metric collection on reconfiguration (#177) * Start and stop metric collection on Start/Stop op * Fix a typo * [MINOR] Use actual id for StartOp in plan execution (#178) * [MINOR] Print a plan when executing it * [MINOR] Fix table control state in master (#180) * [ET-96] Sync task stop in StopOp #183 * [MINOR]Track memory usage in bulk dataloader (#184) * [ET-185] Expose block-level access API for local blocks (#186) * commit * Minor fix * [MINOR] Refine table-access test (#189) * Make sync more safe * [ET-176] Implements multiput() interface (#188) * [ET-175] Separate table load/init and enable bulk-loading of key-value data (#190) * [ET-176] Implement multi-update on ET (#191) * [MINOR] Change ETMsg's Avro schema to fix an Avro problem (#192) * [MINOR] Make MetricCollector reusable (#193) * Use REEF's default DriverLauncher (#198) * [MINOR] Refine example apps (#194) * [ET-195, 196] Refine logics in controlling tables in executors (#197) * [MINOR] Fix ownership cache update method (#199) * [MINOR] Fix errors in MigrationManager (#200) * [MINOR] Refines code (#207) * [ET-210] Use G1 GC in Evaluator processes (#211) * [MINOR} Use catchable executors (#203) * [ET-84] Specify that put/update do not accept null value and throw NPE (#208) * [ET-204] Accumulate ownership updates while initializing tables in executors (#205) * [ET-137] Maintain TableComponents when ongoing ops exist (#206) * [ET-201] Ownership request mechanism (#212) * [ET-213] Handle failed table access request msg (#214) * [MINOR] Separate interface/implementation of AllocatedTable (#217) * commit * Update comment * private annotation * [MINOR] Improve concurrency (#218) * Use a thread pool for handling table-control messages * remove unnecessary synchronized block * Use user-level thread pool for handling all msgs * Fix a style error * [ET-162] Table checkpoint and restoration (#216) * AllocatedTable interface * checkpoint * restore from checkpoint * minor * Add comments. Refine code * refine code * remove separate package * minor * synchronize * nit * implement chkp commit * Option to use yarn runtime * Add todo comment #215 * Improve comment * Improve comment * minor improve * nit * resolve comments * Fix a bug * Add more logs * minor * [MINOR] Make chkp path configurable * [MINOR] Fix a concurrency bug by #218 (#222) * minor change log (#223) * [ET-219] Chunk-unit migration (#220) * commit * Use streaming codec for table codecs * handle empty block. handle concurrent migration in different tables * Make ChunkSize configurable * Minor fix * [ET-215] Use streaming codec for Table's key/value codec #225 * [MINOR] Instantiate Tables as a service component (#229) * [ET-226] Prevent data access to a block while checkpointing it (#228) * [MINOR] Fully parallelize the plan execution #227 * [MINOR] Op tracker in op handler (#230) * op tracker in op handler * clean up logic * comment * update comment * [MINOR] Remove checkpoint done log from evaluators (#232) * [ET-233] Checkpoint with sampling (#234) * checkpoint sampling * update comments * Minor rename * Fix checkstyle violation * compile et * Resolve the relativePath field in ET's pom
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
#163 adds a Random Optimizer. This Random Optimizer is pretty demanding. We should add a simpler example as well.
Quoting @bgchun:
The text was updated successfully, but these errors were encountered: