Merge branch 'cg_merges_20130513' into bfgminer

README.scrypt

@@ -1,12 +1,11 @@
 If you wish to donate to the author of scrypt support, Con Kolivas, please send
 your donations to:
 
-Bitcoin : 15qSxP1SQcUX3o4nhkfdbgyoWEFMomJ4rZ
+Bitcoin : 15qSxP1SQcUX3o4nhkfdbgyoWEFMomJ4rZ (preferred)
 Litecoin: Lc8TWMiKM7gRUrG8VB8pPNP1Yvt1SGZnoH
 
 ---
 
-
 Scrypt mining for GPU is completely different to sha256 used for bitcoin
 mining. It has very different requirements to bitcoin mining and is a
 lot more complicated to get working well. Note that it is a ram dependent
@@ -14,11 +13,10 @@ workload, and requires you to have enough system ram as well as fast enough
 GPU ram. If you have less system ram than your GPU has, it may not be possible
 to mine at any reasonable rate.
 
-There are 5 main parameters to tuning scrypt, 2 of which you MUST set, and
-the others are optional for further fine tuning. When you start scrypt mining
-with the --scrypt option, BFGMiner will fail IN RANDOM WAYS. They are all due
-to parameters being outside what the GPU can cope with. Not giving BFGMiner a
-hint as to your GPU type, it will hardly ever perform well.
+There are 5 main parameters to tuning scrypt, all of which are optional for
+further fine tuning. When you start scrypt mining with the --scrypt option,
+BFGMiner will fail IN RANDOM WAYS. They are all due to parameters being outside
+what the GPU can cope with.
 
 NOTE that if it does not fail at startup, the presence of hardware errors (HW)
 are a sure sign that you have set the parameters too high.
@@ -44,13 +42,31 @@ lines are in the .bat before starting BFGMiner:
 setx GPU_MAX_ALLOC_PERCENT 100
 setx GPU_USE_SYNC_OBJECTS 1
 
+--intensity XX (-I XX)
+
+Just like in Bitcoin mining, scrypt mining takes an intensity, however the
+scale goes from 0 to 20 to mimic the "Aggression" used in mtrlt's reaper. The
+reason this is crucial is that too high an intensity can actually be
+disastrous with scrypt because it CAN run out of ram. High intensities
+start writing over the same ram and it is highly dependent on the GPU, but they
+can start actually DECREASING your hashrate, or even worse, start producing
+garbage with HW errors skyrocketing. Note that if you do NOT specify an
+intensity, cgminer uses dynamic mode which is designed to minimise the harm
+to a running desktop and performance WILL be poor. The lower limit to intensity
+with scrypt is usually 8 and cgminer will prevent it going too low.
+SUMMARY: Setting this for reasonable hashrates is mandatory.
+
 --shaders XXX
 
 is a new option where you tell BFGMiner how many shaders your GPU has. This
 helps BFGMiner try to choose some meaningful baseline parameters. Use this table
 below to determine how many shaders your GPU has, and note that there are some
 variants of these cards, and Nvidia shaders are much much lower and virtually
-pointless trying to mine on.
+pointless trying to mine on. If this is not set, cgminer will query the
+device for how much memory it supports and will try to set a value based on
+that.
+SUMMARY: This will get you started but fine tuning for optimal performance is
+required.
 
 GPU  Shaders
 7750 512
@@ -84,35 +100,24 @@ These are only used as a rough guide for BFGMiner, and it is rare that this is
 all you will need to set.
 
 
---intensity XX
-
-Just like in Bitcoin mining, scrypt mining takes an intensity, however the
-scale goes from 0 to 20 to mimic the "Aggression" used in mtrlt's reaper. The
-reason this is crucial is that too high an intensity can actually be
-disastrous with scrypt because it CAN run out of ram. Intensities over 13
-start writing over the same ram and it is highly dependent on the GPU, but they
-can start actually DECREASING your hashrate, or even worse, start producing
-garbage with HW errors skyrocketing. The low level detail is that intensity is
-only guaranteed up to the power of 2 that most closely matches the thread
-concurrency. i.e. a thread concurrency of 6144 has 8192 as the nearest power
-of two above it, thus as 2^13=8192, that is an intensity of 13.
-
-
 Optional parameters to tune:
 -g, --thread-concurrency, --lookup-gap
 
--g:
-Once you have found the optimal shaders and intensity, you can start increasing
-the -g value till BFGMiner fails to start. Rarely will you be able to go over
-about -g 4 and each increase in -g only increases hashrate slightly.
-
 --thread-concurrency:
 This tunes the optimal size of work that scrypt can do. It is internally tuned
 by BFGMiner to be the highest reasonable multiple of shaders that it can
 allocate on your GPU. Ideally it should be a multiple of your shader count.
 vliw5 architecture (R5XXX) would be best at 5x shaders, while VLIW4 (R6xxx and
 R7xxx) are best at 4x. Setting thread concurrency overrides anything you put
-into --shaders.
+into --shaders and is ultimately a BETTER way to tune performance.
+SUMMARY: Spend lots of time finding the highest value that your device likes
+and increases hashrate.
+
+-g:
+Once you have found the optimal shaders and intensity, you can start increasing
+the -g value till BFGMiner fails to start. This is really only of value if you
+want to run low intensities as you will be unable to run more than 1.
+SUMMARY: Don't touch this.
 
 --lookup-gap
 This tunes a compromise between ram usage and performance. Performance peaks
@@ -120,6 +125,18 @@ at a gap of 2, but increasing the gap can save you some GPU ram, but almost
 always at the cost of significant loss of hashrate. Setting lookup gap
 overrides the default of 2, but BFGMiner will use the --shaders value to choose
 a thread-concurrency if you haven't chosen one.
+SUMMARY: Don't touch this.
+
+
+Related parameters:
+--worksize XX (-w XX)
+Has a minor effect, should be a multiple of 64 up to 256 maximum.
+SUMMARY: Worth playing with once everything else has been tried but will
+probably do nothing.
+
+--vectors XX (-v XX)
+Vectors are NOT used by the scrypt mining kernel.
+SUMMARY: Does nothing.
 
 
 Overclocking for scrypt mining:
@@ -131,20 +148,23 @@ Second, absolute engine clock speeds do NOT correlate with hashrate. The ratio
 of engine clock speed to memory matters, so if you set your memory to the
 default value, and then start overclocking as you are running it, you should
 find a sweet spot where the hashrate peaks and then it might actually drop if
-you increase the engine clock speed further. Unless you wish to run with a
-dynamic intensity, do not go over 13 without testing it while it's running to
-see that it increases hashrate AND utility WITHOUT increasing your HW errors.
-
+you increase the engine clock speed further.
 
-Suggested values for 7970 for example:
-export GPU_MAX_ALLOC_PERCENT=100
---thread-concurrency 8192 -g 4 --gpu-engine 1135 --gpu-memclock 1375
+Third, the combination of motherboard, CPU and system ram ALSO makes a
+difference, so values that work for a GPU on one system may not work for the
+same GPU on a different system. A decent amount of system ram is actually
+required for scrypt mining, and 4GB is suggested.
 
+Finally, the power consumption while mining at high engine clocks, very high
+memory clocks can be far in excess of what you might imagine.
+For example, a 7970 running with the following settings:
+--thread-concurrency 22392 --gpu-engine 1135 --gpu-memclock 1890
+was using 305W!
 
 ---
 
 If you wish to donate to the author of scrypt support, Con Kolivas, please send
 your donations to:
 
-Bitcoin : 15qSxP1SQcUX3o4nhkfdbgyoWEFMomJ4rZ
+Bitcoin : 15qSxP1SQcUX3o4nhkfdbgyoWEFMomJ4rZ (preferred)
 Litecoin: Lc8TWMiKM7gRUrG8VB8pPNP1Yvt1SGZnoH

api.c

@@ -122,7 +122,6 @@ char *WSAErrorMsg(void) {
 	return &(WSAbuf[0]);
 }
 #endif
-static SOCKETTYPE sock = INVSOCK;
 
 static const char *UNAVAILABLE = " - API will not be available";
 static const char *INVAPIGROUPS = "Invalid --api-groups parameter";
@@ -1172,7 +1171,7 @@ static int numpgas()
 	int count = 0;
 	int i;
 
-	mutex_lock(&devices_lock);
+	rd_lock(&devices_lock);
 	for (i = 0; i < total_devices; i++) {
 #ifdef HAVE_OPENCL
 		if (devices[i]->drv == &opencl_api)
@@ -1184,7 +1183,7 @@ static int numpgas()
 #endif
 		++count;
 	}
-	mutex_unlock(&devices_lock);
+	rd_unlock(&devices_lock);
 	return count;
 }
 
@@ -1193,7 +1192,7 @@ static int pgadevice(int pgaid)
 	int count = 0;
 	int i;
 
-	mutex_lock(&devices_lock);
+	rd_lock(&devices_lock);
 	for (i = 0; i < total_devices; i++) {
 #ifdef HAVE_OPENCL
 		if (devices[i]->drv == &opencl_api)
@@ -1208,12 +1207,12 @@ static int pgadevice(int pgaid)
 			goto foundit;
 	}
 
-	mutex_unlock(&devices_lock);
+	rd_unlock(&devices_lock);
 	return -1;
 
 foundit:
 
-	mutex_unlock(&devices_lock);
+	rd_unlock(&devices_lock);
 	return i;
 }
 #endif
@@ -1487,14 +1486,14 @@ static void devdetail_an(struct io_data *io_data, struct cgpu_info *cgpu, bool i
 
 	cgpu->utility = cgpu->accepted / ( total_secs ? total_secs : 1 ) * 60;
 
-	mutex_lock(&devices_lock);
+	rd_lock(&devices_lock);
 	for (i = 0; i < total_devices; ++i) {
 		if (devices[i] == cgpu)
 			break;
 		if (cgpu->devtype == devices[i]->devtype)
 			++n;
 	}
-	mutex_unlock(&devices_lock);
+	rd_unlock(&devices_lock);
 
 	root = api_add_int(root, (char*)cgpu->devtype, &n, true);
 	root = api_add_device_identifier(root, cgpu);
@@ -1521,14 +1520,14 @@ static void devstatus_an(struct io_data *io_data, struct cgpu_info *cgpu, bool i
 
 	cgpu->utility = cgpu->accepted / ( total_secs ? total_secs : 1 ) * 60;
 
-	mutex_lock(&devices_lock);
+	rd_lock(&devices_lock);
 	for (i = 0; i < total_devices; ++i) {
 		if (devices[i] == cgpu)
 			break;
 		if (cgpu->devtype == devices[i]->devtype)
 			++n;
 	}
-	mutex_unlock(&devices_lock);
+	rd_unlock(&devices_lock);
 
 	root = api_add_int(root, (char*)cgpu->devtype, &n, true);
 	root = api_add_device_identifier(root, cgpu);
@@ -2184,13 +2183,16 @@ static void switchpool(struct io_data *io_data, __maybe_unused SOCKETTYPE c, cha
 	}
 
 	id = atoi(param);
+	cg_rlock(&control_lock);
 	if (id < 0 || id >= total_pools) {
+		cg_runlock(&control_lock);
 		message(io_data, MSG_INVPID, id, NULL, isjson);
 		return;
 	}
 
 	pool = pools[id];
 	pool->enabled = POOL_ENABLED;
+	cg_runlock(&control_lock);
 	switch_pools(pool);
 
 	message(io_data, MSG_SWITCHP, id, NULL, isjson);
@@ -2882,7 +2884,7 @@ static void minecoin(struct io_data *io_data, __maybe_unused SOCKETTYPE c, __may
 #endif
 		root = api_add_const(root, "Hash Method", SHA256STR, false);
 
-	mutex_lock(&ch_lock);
+	cg_rlock(&ch_lock);
 	if (current_fullhash && *current_fullhash) {
 		root = api_add_timeval(root, "Current Block Time", &block_timeval, true);
 		root = api_add_string(root, "Current Block Hash", current_fullhash, true);
@@ -2891,7 +2893,7 @@ static void minecoin(struct io_data *io_data, __maybe_unused SOCKETTYPE c, __may
 		root = api_add_timeval(root, "Current Block Time", &t, true);
 		root = api_add_const(root, "Current Block Hash", BLANK, false);
 	}
-	mutex_unlock(&ch_lock);
+	cg_runlock(&ch_lock);
 
 	root = api_add_bool(root, "LP", &have_longpoll, false);
 	root = api_add_diff(root, "Network Difficulty", &current_diff, true);
@@ -3240,8 +3242,7 @@ static void checkcommand(struct io_data *io_data, __maybe_unused SOCKETTYPE c, c
 static void send_result(struct io_data *io_data, SOCKETTYPE c, bool isjson)
 {
 	char buf[SOCKBUFSIZ + sizeof(JSON_CLOSE) + sizeof(JSON_END)];
-	int len;
-	int n;
+	int count, res, tosend, len, n;
 
 	strcpy(buf, io_data->ptr);
 
@@ -3256,28 +3257,62 @@ static void send_result(struct io_data *io_data, SOCKETTYPE c, bool isjson)
 	}
 
 	len = strlen(buf);
+	tosend = len+1;
 
-	applog(LOG_DEBUG, "API: send reply: (%d) '%.10s%s'", len+1, buf, len > 10 ? "..." : BLANK);
+	applog(LOG_DEBUG, "API: send reply: (%d) '%.10s%s'", tosend, buf, len > 10 ? "..." : BLANK);
 
-	// ignore failure - it's closed immediately anyway
-	n = send(c, buf, len+1, 0);
+	count = 0;
+	while (count++ < 5 && tosend > 0) {
+		// allow 50ms per attempt
+		struct timeval timeout = {0, 50000};
+		fd_set wd;
+
+		FD_ZERO(&wd);
+		FD_SET(c, &wd);
+		if ((res = select(c + 1, NULL, &wd, NULL, &timeout)) < 1) {
+			applog(LOG_WARNING, "API: send select failed (%d)", res);
+			return;
+		}
 
-	if (SOCKETFAIL(n))
-		applog(LOG_WARNING, "API: send failed: %s", SOCKERRMSG);
-	else
-		applog(LOG_DEBUG, "API: sent %d", n);
+		n = send(c, buf, tosend, 0);
+
+		if (SOCKETFAIL(n)) {
+			if (errno == EAGAIN || errno == EWOULDBLOCK)
+				continue;
+
+			applog(LOG_WARNING, "API: send (%d) failed: %s", tosend, SOCKERRMSG);
+
+			return;
+		} else {
+			if (count <= 1) {
+				if (n == tosend)
+					applog(LOG_DEBUG, "API: sent all of %d first go", tosend);
+				else
+					applog(LOG_DEBUG, "API: sent %d of %d first go", n, tosend);
+			} else {
+				if (n == tosend)
+					applog(LOG_DEBUG, "API: sent all of remaining %d (count=%d)", tosend, count);
+				else
+					applog(LOG_DEBUG, "API: sent %d of remaining %d (count=%d)", n, tosend, count);
+			}
+
+			tosend -= n;
+		}
+	}
 }
 
 static void tidyup(__maybe_unused void *arg)
 {
 	mutex_lock(&quit_restart_lock);
 
+	SOCKETTYPE *apisock = (SOCKETTYPE *)arg;
+
 	bye = true;
 
-	if (sock != INVSOCK) {
-		shutdown(sock, SHUT_RDWR);
-		CLOSESOCKET(sock);
-		sock = INVSOCK;
+	if (*apisock != INVSOCK) {
+		shutdown(*apisock, SHUT_RDWR);
+		CLOSESOCKET(*apisock);
+		*apisock = INVSOCK;
 	}
 
 	if (ipaccess != NULL) {
@@ -3598,6 +3633,11 @@ void api(int api_thr_id)
 	bool did;
 	int i;
 
+	SOCKETTYPE *apisock;
+
+	apisock = malloc(sizeof(*apisock));
+	*apisock = INVSOCK;
+
 	if (!opt_api_listen) {
 		applog(LOG_DEBUG, "API not running%s", UNAVAILABLE);
 		return;
@@ -3607,7 +3647,7 @@ void api(int api_thr_id)
 
 	mutex_init(&quit_restart_lock);
 
-	pthread_cleanup_push(tidyup, NULL);
+	pthread_cleanup_push(tidyup, (void *)apisock);
 	my_thr_id = api_thr_id;
 
 	setup_groups();
@@ -3625,8 +3665,8 @@ void api(int api_thr_id)
 	 * to ensure curl has already called WSAStartup() in windows */
 	nmsleep(opt_log_interval*1000);
 
-	sock = socket(AF_INET, SOCK_STREAM, 0);
-	if (sock == INVSOCK) {
+	*apisock = socket(AF_INET, SOCK_STREAM, 0);
+	if (*apisock == INVSOCK) {
 		applog(LOG_ERR, "API1 initialisation failed (%s)%s", SOCKERRMSG, UNAVAILABLE);
 		return;
 	}
@@ -3651,7 +3691,7 @@ void api(int api_thr_id)
 	// another program has it open - which is what we want
 	int optval = 1;
 	// If it doesn't work, we don't really care - just show a debug message
-	if (SOCKETFAIL(setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (void *)(&optval), sizeof(optval))))
+	if (SOCKETFAIL(setsockopt(*apisock, SOL_SOCKET, SO_REUSEADDR, (void *)(&optval), sizeof(optval))))
 		applog(LOG_DEBUG, "API setsockopt SO_REUSEADDR failed (ignored): %s", SOCKERRMSG);
 #else
 	// On windows a 2nd program can bind to a port>1024 already in use unless
@@ -3663,7 +3703,7 @@ void api(int api_thr_id)
 	bound = 0;
 	bindstart = time(NULL);
 	while (bound == 0) {
-		if (SOCKETFAIL(bind(sock, (struct sockaddr *)(&serv), sizeof(serv)))) {
+		if (SOCKETFAIL(bind(*apisock, (struct sockaddr *)(&serv), sizeof(serv)))) {
 			binderror = SOCKERRMSG;
 			if ((time(NULL) - bindstart) > 61)
 				break;
@@ -3680,25 +3720,25 @@ void api(int api_thr_id)
 		return;
 	}
 
-	if (SOCKETFAIL(listen(sock, QUEUE))) {
+	if (SOCKETFAIL(listen(*apisock, QUEUE))) {
 		applog(LOG_ERR, "API3 initialisation failed (%s)%s", SOCKERRMSG, UNAVAILABLE);
-		CLOSESOCKET(sock);
+		CLOSESOCKET(*apisock);
 		return;
 	}
 
 	if (opt_api_allow)
-		applog(LOG_WARNING, "API running in IP access mode on port %d", port);
+		applog(LOG_WARNING, "API running in IP access mode on port %d (%d)", port, *apisock);
 	else {
 		if (opt_api_network)
-			applog(LOG_WARNING, "API running in UNRESTRICTED read access mode on port %d", port);
+			applog(LOG_WARNING, "API running in UNRESTRICTED read access mode on port %d (%d)", port, *apisock);
 		else
-			applog(LOG_WARNING, "API running in local read access mode on port %d", port);
+			applog(LOG_WARNING, "API running in local read access mode on port %d (%d)", port, *apisock);
 	}
 
 	while (!bye) {
 		clisiz = sizeof(cli);
-		if (SOCKETFAIL(c = accept(sock, (struct sockaddr *)(&cli), &clisiz))) {
-			applog(LOG_ERR, "API failed (%s)%s", SOCKERRMSG, UNAVAILABLE);
+		if (SOCKETFAIL(c = accept(*apisock, (struct sockaddr *)(&cli), &clisiz))) {
+			applog(LOG_ERR, "API failed (%s)%s (%d)", SOCKERRMSG, UNAVAILABLE, *apisock);
 			goto die;
 		}
 

deviceapi.c

@@ -597,7 +597,7 @@ bool add_cgpu(struct cgpu_info*cgpu)
 	strcpy(cgpu->proc_repr, cgpu->dev_repr);
 	sprintf(cgpu->proc_repr_ns, "%s%u", cgpu->drv->name, cgpu->device_id);
 	
-	mutex_lock(&devices_lock);
+	wr_lock(&devices_lock);
 	
 	devices = realloc(devices, sizeof(struct cgpu_info *) * (total_devices + lpcount + 1));
 	devices[total_devices++] = cgpu;
@@ -631,7 +631,7 @@ bool add_cgpu(struct cgpu_info*cgpu)
 		cgpu->threads -= (tpp * (lpcount - 1));
 	}
 	
-	mutex_unlock(&devices_lock);
+	wr_unlock(&devices_lock);
 	
 	return true;
 }

driver-opencl.c

@@ -1441,8 +1441,14 @@ static void opencl_detect()
 	if (!nDevs)
 		return;
 
-	if (opt_g_threads == -1)
-		opt_g_threads = 2;
+	/* If opt_g_threads is not set, use default 1 thread on scrypt and
+	 * 2 for regular mining */
+	if (opt_g_threads == -1) {
+		if (opt_scrypt)
+			opt_g_threads = 1;
+		else
+			opt_g_threads = 2;
+	}
 
 #ifdef HAVE_SENSORS
 	struct opencl_device_data *data;

miner.c

@@ -210,16 +210,15 @@ static int api_thr_id;
 static int total_control_threads;
 
 pthread_mutex_t hash_lock;
-static pthread_mutex_t qd_lock;
 static pthread_mutex_t *stgd_lock;
 pthread_mutex_t console_lock;
-pthread_mutex_t ch_lock;
+cglock_t ch_lock;
 static pthread_rwlock_t blk_lock;
 static pthread_mutex_t sshare_lock;
 
 pthread_rwlock_t netacc_lock;
-pthread_mutex_t mining_thr_lock;
-pthread_mutex_t devices_lock;
+pthread_rwlock_t mining_thr_lock;
+pthread_rwlock_t devices_lock;
 
 static pthread_mutex_t lp_lock;
 static pthread_cond_t lp_cond;
@@ -232,7 +231,7 @@ double total_mhashes_done;
 static struct timeval total_tv_start, total_tv_end;
 static struct timeval miner_started;
 
-pthread_mutex_t control_lock;
+cglock_t control_lock;
 pthread_mutex_t stats_lock;
 
 static pthread_mutex_t submitting_lock;
@@ -267,9 +266,12 @@ const
 bool curses_active;
 
 static char current_block[40];
+
+/* Protected by ch_lock */
 static char *current_hash;
 static uint32_t current_block_id;
 char *current_fullhash;
+
 static char datestamp[40];
 static char blocktime[32];
 struct timeval block_timeval;
@@ -418,9 +420,9 @@ struct thr_info *get_thread(int thr_id)
 {
 	struct thr_info *thr;
 
-	mutex_lock(&mining_thr_lock);
+	rd_lock(&mining_thr_lock);
 	thr = mining_thr[thr_id];
-	mutex_unlock(&mining_thr_lock);
+	rd_unlock(&mining_thr_lock);
 	return thr;
 }
 
@@ -435,9 +437,9 @@ struct cgpu_info *get_devices(int id)
 {
 	struct cgpu_info *cgpu;
 
-	mutex_lock(&devices_lock);
+	rd_lock(&devices_lock);
 	cgpu = devices[id];
-	mutex_unlock(&devices_lock);
+	rd_unlock(&devices_lock);
 	return cgpu;
 }
 
@@ -497,6 +499,7 @@ struct pool *add_pool(void)
 	mutex_init(&pool->pool_lock);
 	if (unlikely(pthread_cond_init(&pool->cr_cond, NULL)))
 		quit(1, "Failed to pthread_cond_init in add_pool");
+	cglock_init(&pool->data_lock);
 	mutex_init(&pool->stratum_lock);
 	INIT_LIST_HEAD(&pool->curlring);
 	pool->swork.transparency_time = (time_t)-1;
@@ -542,9 +545,9 @@ struct pool *current_pool(void)
 {
 	struct pool *pool;
 
-	mutex_lock(&control_lock);
+	cg_rlock(&control_lock);
 	pool = currentpool;
-	mutex_unlock(&control_lock);
+	cg_runlock(&control_lock);
 	return pool;
 }
 
@@ -1790,9 +1793,9 @@ static struct work *make_work(void)
 
 	if (unlikely(!work))
 		quit(1, "Failed to calloc work in make_work");
-	mutex_lock(&control_lock);
+	cg_wlock(&control_lock);
 	work->id = total_work++;
-	mutex_unlock(&control_lock);
+	cg_wunlock(&control_lock);
 	return work;
 }
 
@@ -1861,12 +1864,12 @@ void have_block_height(uint32_t block_id, uint32_t blkheight)
 	if (known_blkheight == blkheight)
 		return;
 	applog(LOG_DEBUG, "Learned that block id %08" PRIx32 " is height %" PRIu32, be32toh(block_id), blkheight);
-	mutex_lock(&ch_lock);
+	cg_wlock(&ch_lock);
 	known_blkheight = blkheight;
 	known_blkheight_blkid = block_id;
 	if (block_id == current_block_id)
 		__update_block_title(NULL);
-	mutex_unlock(&ch_lock);
+	cg_wunlock(&ch_lock);
 }
 
 static bool work_decode(struct pool *pool, struct work *work, json_t *val)
@@ -2487,8 +2490,10 @@ static void curses_print_status(void)
 			pool->sockaddr_url, pool->diff, have_longpoll ? "": "out", pool->rpc_user);
 	}
 	wclrtoeol(statuswin);
+	cg_rlock(&ch_lock);
 	mvwprintw(statuswin, 5, 0, " Block: %s  Diff:%s (%s)  Started: %s",
 		  current_hash, block_diff, net_hashrate, blocktime);
+	cg_runlock(&ch_lock);
 	mvwhline(statuswin, 6, 0, '-', 80);
 	mvwhline(statuswin, statusy - 1, 0, '-', 80);
 	mvwprintw(statuswin, devcursor - 1, 1, "[P]ool management %s[S]ettings [D]isplay options [Q]uit",
@@ -2908,7 +2913,7 @@ static uint64_t share_diff(const struct work *work)
 	bool new_best = false;
 
 	ret = target_diff(work->hash);
-	mutex_lock(&control_lock);
+	cg_wlock(&control_lock);
 	if (unlikely(ret > best_diff)) {
 		new_best = true;
 		best_diff = ret;
@@ -2916,7 +2921,7 @@ static uint64_t share_diff(const struct work *work)
 	}
 	if (unlikely(ret > work->pool->best_diff))
 		work->pool->best_diff = ret;
-	mutex_unlock(&control_lock);
+	cg_wunlock(&control_lock);
 
 	if (unlikely(new_best))
 		applog(LOG_INFO, "New best share: %s", best_share);
@@ -3058,6 +3063,18 @@ out:
 	return rc;
 }
 
+/* Specifies whether we can use this pool for work or not. */
+static bool pool_unworkable(struct pool *pool)
+{
+	if (pool->idle)
+		return true;
+	if (pool->enabled != POOL_ENABLED)
+		return true;
+	if (pool->has_stratum && !pool->stratum_active)
+		return true;
+	return false;
+}
+
 /* In balanced mode, the amount of diff1 solutions per pool is monitored as a
  * rolling average per 10 minutes and if pools start getting more, it biases
  * away from them to distribute work evenly. The share count is reset to the
@@ -3071,7 +3088,7 @@ static struct pool *select_balanced(struct pool *cp)
 	for (i = 0; i < total_pools; i++) {
 		struct pool *pool = pools[i];
 
-		if (pool->idle || pool->enabled != POOL_ENABLED)
+		if (pool_unworkable(pool))
 			continue;
 		if (pool->shares < lowest) {
 			lowest = pool->shares;
@@ -3091,6 +3108,7 @@ static inline struct pool *select_pool(bool lagging)
 {
 	static int rotating_pool = 0;
 	struct pool *pool, *cp;
+	int tested;
 
 	cp = current_pool();
 
@@ -3106,14 +3124,19 @@ retry:
 	else
 		pool = NULL;
 
-	while (!pool) {
+	/* Try to find the first pool in the rotation that is usable */
+	tested = 0;
+	while (!pool && tested++ < total_pools) {
 		if (++rotating_pool >= total_pools)
 			rotating_pool = 0;
 		pool = pools[rotating_pool];
-		if ((!pool->idle && pool->enabled == POOL_ENABLED) || pool == cp)
+		if (!pool_unworkable(pool))
 			break;
 		pool = NULL;
 	}
+	/* If still nothing is usable, use the current pool */
+	if (!pool)
+		pool = cp;
 
 have_pool:
 	if (cp != pool)
@@ -3848,10 +3871,10 @@ bool stale_work(struct work *work, bool share)
 		}
 
 		same_job = true;
-		mutex_lock(&pool->pool_lock);
+		cg_rlock(&pool->data_lock);
 		if (strcmp(work->job_id, pool->swork.job_id))
 			same_job = false;
-		mutex_unlock(&pool->pool_lock);
+		cg_runlock(&pool->data_lock);
 		if (!same_job) {
 			applog(LOG_DEBUG, "Work stale due to stratum job_id mismatch");
 			return true;
@@ -4135,7 +4158,7 @@ static void *submit_work_thread(__maybe_unused void *userdata)
 		{
 			struct pool *pool = sws->work->pool;
 			int fd = pool->sock;
-			if (fd == INVSOCK || (!pool->stratum_auth) || !pool->stratum_notify)
+			if (fd == INVSOCK || (!pool->stratum_init) || !pool->stratum_notify)
 				continue;
 			FD_SET(fd, &wfds);
 			set_maxfd(&maxfd, fd);
@@ -4159,11 +4182,11 @@ static void *submit_work_thread(__maybe_unused void *userdata)
 			int fd = pool->sock;
 			bool sessionid_match;
 			
-			mutex_lock(&pool->pool_lock);
+			cg_rlock(&pool->data_lock);
 			// NOTE: cgminer only does this check on retries, but BFGMiner does it for even the first/normal submit; therefore, it needs to be such that it always is true on the same connection regardless of session management
 			// NOTE: Worst case scenario for a false positive: the pool rejects it as H-not-zero
 			sessionid_match = (!pool->nonce1) || !strcmp(work->nonce1, pool->nonce1);
-			mutex_unlock(&pool->pool_lock);
+			cg_runlock(&pool->data_lock);
 			if (!sessionid_match)
 			{
 				applog(LOG_DEBUG, "No matching session id for resubmitting stratum share");
@@ -4179,7 +4202,7 @@ next_write_sws_del:
 				continue;
 			}
 			
-			if (fd == INVSOCK || (!pool->stratum_auth) || (!pool->stratum_notify) || !FD_ISSET(fd, &wfds)) {
+			if (fd == INVSOCK || (!pool->stratum_init) || (!pool->stratum_notify) || !FD_ISSET(fd, &wfds)) {
 next_write_sws:
 				// TODO: Check if stale, possibly discard etc
 				swsp = &sws->next;
@@ -4388,12 +4411,24 @@ void validate_pool_priorities(void)
 	}
 }
 
+static void clear_pool_work(struct pool *pool);
+
+/* Specifies whether we can switch to this pool or not. */
+static bool pool_unusable(struct pool *pool)
+{
+	if (pool->idle)
+		return true;
+	if (pool->enabled != POOL_ENABLED)
+		return true;
+	return false;
+}
+
 void switch_pools(struct pool *selected)
 {
 	struct pool *pool, *last_pool;
 	int i, pool_no, next_pool;
 
-	mutex_lock(&control_lock);
+	cg_wlock(&control_lock);
 	last_pool = currentpool;
 	pool_no = currentpool->pool_no;
 
@@ -4416,10 +4451,10 @@ void switch_pools(struct pool *selected)
 		case POOL_LOADBALANCE:
 			for (i = 0; i < total_pools; i++) {
 				pool = priority_pool(i);
-				if (!pool->idle && pool->enabled == POOL_ENABLED) {
-					pool_no = pool->pool_no;
-					break;
-				}
+				if (pool_unusable(pool))
+					continue;
+				pool_no = pool->pool_no;
+				break;
 			}
 			break;
 		/* Both of these simply increment and cycle */
@@ -4436,10 +4471,10 @@ void switch_pools(struct pool *selected)
 				if (next_pool >= total_pools)
 					next_pool = 0;
 				pool = pools[next_pool];
-				if (!pool->idle && pool->enabled == POOL_ENABLED) {
-					pool_no = next_pool;
-					break;
-				}
+				if (pool_unusable(pool))
+					continue;
+				pool_no = next_pool;
+				break;
 			}
 			break;
 		default:
@@ -4448,7 +4483,7 @@ void switch_pools(struct pool *selected)
 
 	currentpool = pools[pool_no];
 	pool = currentpool;
-	mutex_unlock(&control_lock);
+	cg_wunlock(&control_lock);
 
 	/* Set the lagging flag to avoid pool not providing work fast enough
 	 * messages in failover only mode since  we have to get all fresh work
@@ -4461,6 +4496,8 @@ void switch_pools(struct pool *selected)
 		pool->block_id = 0;
 		if (pool_strategy != POOL_LOADBALANCE && pool_strategy != POOL_BALANCE) {
 			applog(LOG_WARNING, "Switching to pool %d %s", pool->pool_no, pool->rpc_url);
+			if (pool->last_work_copy || pool->has_stratum || opt_fail_only)
+				clear_pool_work(last_pool);
 		}
 	}
 
@@ -4541,7 +4578,7 @@ static void restart_threads(void)
 	/* Discard staged work that is now stale */
 	discard_stale();
 
-	mutex_lock(&mining_thr_lock);
+	rd_lock(&mining_thr_lock);
 	
 	for (i = 0; i < mining_threads; i++)
 	{
@@ -4555,7 +4592,7 @@ static void restart_threads(void)
 		notifier_wake(thr->work_restart_notifier);
 	}
 	
-	mutex_unlock(&mining_thr_lock);
+	rd_unlock(&mining_thr_lock);
 }
 
 static char *blkhashstr(unsigned char *hash)
@@ -4569,26 +4606,20 @@ static char *blkhashstr(unsigned char *hash)
 static void set_curblock(char *hexstr, unsigned char *hash)
 {
 	unsigned char hash_swap[32];
-	char *old_hash;
 
 	current_block_id = ((uint32_t*)hash)[0];
 	strcpy(current_block, hexstr);
 	swap256(hash_swap, hash);
 	swap32tole(hash_swap, hash_swap, 32 / 4);
 
-	/* Don't free current_hash directly to avoid dereferencing when read
-	 * elsewhere - and update block_timeval inside the same lock */
-	mutex_lock(&ch_lock);
+	cg_wlock(&ch_lock);
 	gettimeofday(&block_timeval, NULL);
 	__update_block_title(hash_swap);
-	old_hash = current_fullhash;
+	free(current_fullhash);
 	current_fullhash = bin2hex(hash_swap, 32);
-	free(old_hash);
-	mutex_unlock(&ch_lock);
-
 	get_timestamp(blocktime, &block_timeval);
-
 	applog(LOG_INFO, "New block: %s diff %s (%s)", current_hash, block_diff, net_hashrate);
+	cg_wunlock(&ch_lock);
 }
 
 /* Search to see if this string is from a block that has been seen before */
@@ -5998,7 +6029,7 @@ static void shutdown_stratum(struct pool *pool)
 {
 	// Shut down Stratum as if we never had it
 	pool->stratum_active = false;
-	pool->stratum_auth = false;
+	pool->stratum_init = false;
 	pool->has_stratum = false;
 	shutdown(pool->sock, SHUT_RDWR);
 	free(pool->stratum_url);
@@ -6090,6 +6121,16 @@ static void clear_pool_work(struct pool *pool)
 	mutex_unlock(stgd_lock);
 }
 
+static int cp_prio(void)
+{
+	int prio;
+
+	cg_rlock(&control_lock);
+	prio = currentpool->prio;
+	cg_runlock(&control_lock);
+	return prio;
+}
+
 /* We only need to maintain a secondary pool connection when we need the
  * capacity to get work from the backup pools while still on the primary */
 static bool cnx_needed(struct pool *pool)
@@ -6119,6 +6160,15 @@ static bool cnx_needed(struct pool *pool)
 	if (difftime(time(NULL), pool->last_work_time) < 120)
 		return true;
 
+	/* If the pool has only just come to life and is higher priority than
+	 * the current pool keep the connection open so we can fail back to
+	 * it. */
+	if (pool_strategy == POOL_FAILOVER && pool->prio < cp_prio())
+		return true;
+
+	if (pool_unworkable(cp))
+		return true;
+
 	return false;
 }
 
@@ -6140,9 +6190,9 @@ static bool supports_resume(struct pool *pool)
 {
 	bool ret;
 
-	mutex_lock(&pool->pool_lock);
+	cg_rlock(&pool->data_lock);
 	ret = (pool->sessionid != NULL);
-	mutex_unlock(&pool->pool_lock);
+	cg_runlock(&pool->data_lock);
 	return ret;
 }
 
@@ -6164,6 +6214,7 @@ static void *stratum_thread(void *userdata)
 
 	while (42) {
 		struct timeval timeout;
+		int sel_ret;
 		fd_set rd;
 		char *s;
 
@@ -6197,8 +6248,8 @@ static void *stratum_thread(void *userdata)
 		/* If we fail to receive any notify messages for 2 minutes we
 		 * assume the connection has been dropped and treat this pool
 		 * as dead */
-		if (!sock_full(pool) && select(pool->sock + 1, &rd, NULL, NULL, &timeout) < 1) {
-			applog(LOG_DEBUG, "Stratum select timeout on pool %d", pool->pool_no);
+		if (!sock_full(pool) && (sel_ret = select(pool->sock + 1, &rd, NULL, NULL, &timeout)) < 1) {
+			applog(LOG_DEBUG, "Stratum select failed on pool %d with value %d", pool->pool_no, sel_ret);
 			s = NULL;
 		} else
 			s = recv_line(pool);
@@ -6368,16 +6419,21 @@ retry_stratum:
 		curl_easy_cleanup(curl);
 		
 		/* We create the stratum thread for each pool just after
-		 * successful authorisation. Once the auth flag has been set
+		 * successful authorisation. Once the init flag has been set
 		 * we never unset it and the stratum thread is responsible for
 		 * setting/unsetting the active flag */
-		if (pool->stratum_auth)
-			return pool->stratum_active;
-		if (!(pool->stratum_active ? auth_stratum(pool) : restart_stratum(pool)))
-			return false;
-		init_stratum_thread(pool);
-		detect_algo = 2;
-		return true;
+		bool init = pool_tset(pool, &pool->stratum_init);
+
+		if (!init) {
+			bool ret = initiate_stratum(pool) && auth_stratum(pool);
+
+			if (ret)
+				init_stratum_thread(pool);
+			else
+				pool_tclear(pool, &pool->stratum_init);
+			return ret;
+		}
+		return pool->stratum_active;
 	}
 	else if (pool->has_stratum)
 		shutdown_stratum(pool);
@@ -6483,23 +6539,13 @@ out:
 	return ret;
 }
 
-static inline int cp_prio(void)
-{
-	int prio;
-
-	mutex_lock(&control_lock);
-	prio = currentpool->prio;
-	mutex_unlock(&control_lock);
-	return prio;
-}
-
 static void pool_resus(struct pool *pool)
 {
-	if (pool->prio < cp_prio() && pool_strategy == POOL_FAILOVER) {
+	if (pool_strategy == POOL_FAILOVER && pool->prio < cp_prio()) {
 		applog(LOG_WARNING, "Pool %d %s alive", pool->pool_no, pool->rpc_url);
 		switch_pools(NULL);
 	} else
-		applog(LOG_INFO, "Pool %d %s resumed returning work", pool->pool_no, pool->rpc_url);
+		applog(LOG_INFO, "Pool %d %s alive", pool->pool_no, pool->rpc_url);
 }
 
 static struct work *hash_pop(void)
@@ -6648,11 +6694,15 @@ static void gen_stratum_work(struct pool *pool, struct work *work)
 
 	clean_work(work);
 
-	mutex_lock(&pool->pool_lock);
+	/* Use intermediate lock to update the one pool variable */
+	cg_ilock(&pool->data_lock);
 
 	/* Generate coinbase */
 	work->nonce2 = bin2hex((const unsigned char *)&pool->nonce2, pool->n2size);
 	pool->nonce2++;
+
+	/* Downgrade to a read lock to read off the pool variables */
+	cg_dlock(&pool->data_lock);
 	alloc_len = pool->swork.cb_len;
 	align_len(&alloc_len);
 	coinbase = calloc(alloc_len, 1);
@@ -6701,7 +6751,7 @@ static void gen_stratum_work(struct pool *pool, struct work *work)
 	work->job_id = strdup(pool->swork.job_id);
 	work->nonce1 = strdup(pool->nonce1);
 	work->ntime = strdup(pool->swork.ntime);
-	mutex_unlock(&pool->pool_lock);
+	cg_runlock(&pool->data_lock);
 
 	applog(LOG_DEBUG, "Generated stratum merkle %s", merkle_hash);
 	applog(LOG_DEBUG, "Generated stratum header %s", header);
@@ -6948,6 +6998,166 @@ void mt_disable_start(struct thr_info *mythr)
 	thread_reportout(mythr);
 }
 
+/* Create a hashtable of work items for devices with a queue. The device
+ * driver must have a custom queue_full function or it will default to true
+ * and put only one work item in the queue. Work items should not be removed
+ * from this hashtable until they are no longer in use anywhere. Once a work
+ * item is physically queued on the device itself, the work->queued flag
+ * should be set under cgpu->qlock write lock to prevent it being dereferenced
+ * while still in use. */
+static void fill_queue(struct thr_info *mythr, struct cgpu_info *cgpu, struct device_drv *drv, const int thr_id)
+{
+	thread_reportout(mythr);
+	do {
+		struct work *work = get_work(mythr);
+
+		wr_lock(&cgpu->qlock);
+		HASH_ADD_INT(cgpu->queued_work, id, work);
+		wr_unlock(&cgpu->qlock);
+		/* The queue_full function should be used by the driver to
+		 * actually place work items on the physical device if it
+		 * does have a queue. */
+	} while (drv->queue_full && !drv->queue_full(cgpu));
+}
+
+/* This function is for retrieving one work item from the queued hashtable of
+ * available work items that are not yet physically on a device (which is
+ * flagged with the work->queued bool). Code using this function must be able
+ * to handle NULL as a return which implies there is no work available. */
+struct work *get_queued(struct cgpu_info *cgpu)
+{
+	struct work *work, *tmp, *ret = NULL;
+
+	wr_lock(&cgpu->qlock);
+	HASH_ITER(hh, cgpu->queued_work, work, tmp) {
+		if (!work->queued) {
+			work->queued = true;
+			ret = work;
+			break;
+		}
+	}
+	wr_unlock(&cgpu->qlock);
+
+	return ret;
+}
+
+/* This function is for finding an already queued work item in the
+ * given que hashtable. Code using this function must be able
+ * to handle NULL as a return which implies there is no matching work.
+ * The calling function must lock access to the que if it is required.
+ * The common values for midstatelen, offset, datalen are 32, 64, 12 */
+struct work *__find_work_bymidstate(struct work *que, char *midstate, size_t midstatelen, char *data, int offset, size_t datalen)
+{
+	struct work *work, *tmp, *ret = NULL;
+
+	HASH_ITER(hh, que, work, tmp) {
+		if (work->queued &&
+		    memcmp(work->midstate, midstate, midstatelen) == 0 &&
+		    memcmp(work->data + offset, data, datalen) == 0) {
+			ret = work;
+			break;
+		}
+	}
+
+	return ret;
+}
+
+/* This function is for finding an already queued work item in the
+ * device's queued_work hashtable. Code using this function must be able
+ * to handle NULL as a return which implies there is no matching work.
+ * The common values for midstatelen, offset, datalen are 32, 64, 12 */
+struct work *find_queued_work_bymidstate(struct cgpu_info *cgpu, char *midstate, size_t midstatelen, char *data, int offset, size_t datalen)
+{
+	struct work *ret;
+
+	rd_lock(&cgpu->qlock);
+	ret = __find_work_bymidstate(cgpu->queued_work, midstate, midstatelen, data, offset, datalen);
+	rd_unlock(&cgpu->qlock);
+
+	return ret;
+}
+
+/* This function should be used by queued device drivers when they're sure
+ * the work struct is no longer in use. */
+void work_completed(struct cgpu_info *cgpu, struct work *work)
+{
+	wr_lock(&cgpu->qlock);
+	HASH_DEL(cgpu->queued_work, work);
+	wr_unlock(&cgpu->qlock);
+	free_work(work);
+}
+
+static void flush_queue(struct cgpu_info *cgpu)
+{
+	struct work *work, *tmp;
+	int discarded = 0;
+
+	wr_lock(&cgpu->qlock);
+	HASH_ITER(hh, cgpu->queued_work, work, tmp) {
+		/* Can only discard the work items if they're not physically
+		 * queued on the device. */
+		if (!work->queued) {
+			HASH_DEL(cgpu->queued_work, work);
+			discard_work(work);
+			discarded++;
+		}
+	}
+	wr_unlock(&cgpu->qlock);
+
+	if (discarded)
+		applog(LOG_DEBUG, "Discarded %d queued work items", discarded);
+}
+
+/* This version of hash work is for devices that are fast enough to always
+ * perform a full nonce range and need a queue to maintain the device busy.
+ * Work creation and destruction is not done from within this function
+ * directly. */
+void hash_queued_work(struct thr_info *mythr)
+{
+	const long cycle = opt_log_interval / 5 ? : 1;
+	struct timeval tv_start = {0, 0}, tv_end;
+	struct cgpu_info *cgpu = mythr->cgpu;
+	struct device_drv *drv = cgpu->drv;
+	const int thr_id = mythr->id;
+	int64_t hashes_done = 0;
+
+	while (42) {
+		struct timeval diff;
+		int64_t hashes;
+
+		mythr->work_restart = false;
+
+		fill_queue(mythr, cgpu, drv, thr_id);
+
+		thread_reportin(mythr);
+		hashes = drv->scanwork(mythr);
+		if (unlikely(hashes == -1 )) {
+			applog(LOG_ERR, "%s %d failure, disabling!", drv->name, cgpu->device_id);
+			cgpu->deven = DEV_DISABLED;
+			dev_error(cgpu, REASON_THREAD_ZERO_HASH);
+			mt_disable(mythr);
+		}
+
+		hashes_done += hashes;
+		gettimeofday(&tv_end, NULL);
+		timersub(&tv_end, &tv_start, &diff);
+		if (diff.tv_sec >= cycle) {
+			hashmeter(thr_id, &diff, hashes_done);
+			hashes_done = 0;
+			memcpy(&tv_start, &tv_end, sizeof(struct timeval));
+		}
+
+		if (unlikely(mythr->work_restart)) {
+			flush_queue(cgpu);
+			if (drv->flush_work)
+				drv->flush_work(cgpu);
+		}
+
+		if (unlikely(mythr->pause || cgpu->deven != DEV_ENABLED))
+			mt_disable(mythr);
+	}
+}
+
 void mt_disable_finish(struct thr_info *mythr)
 {
 	struct device_drv *drv = mythr->cgpu->drv;
@@ -7299,16 +7509,22 @@ static void *watchpool_thread(void __maybe_unused *userdata)
 				pool->shares = pool->utility;
 			}
 
-			if ((pool->enabled == POOL_DISABLED || pool->has_stratum) && pool->probed)
+			if (pool->enabled == POOL_DISABLED)
 				continue;
 
+			/* Don't start testing any pools if the test threads
+			 * from startup are still doing their first attempt. */
+			if (unlikely(pool->testing)) {
+				pthread_join(pool->test_thread, NULL);
+				pool->testing = false;
+			}
+
 			/* Test pool is idle once every minute */
 			if (pool->idle && now.tv_sec - pool->tv_idle.tv_sec > 30) {
 				gettimeofday(&pool->tv_idle, NULL);
 				if (pool_active(pool, true) && pool_tclear(pool, &pool->idle))
 					pool_resus(pool);
 			}
-
 		}
 
 		if (pool_strategy == POOL_ROTATE && now.tv_sec - rotate_tv.tv_sec > 60 * opt_rotate_period) {
@@ -7399,10 +7615,10 @@ static void *watchdog_thread(void __maybe_unused *userdata)
 			applog(LOG_WARNING, "Will restart execution as scheduled at %02d:%02d",
 			       schedstart.tm.tm_hour, schedstart.tm.tm_min);
 			sched_paused = true;
-			mutex_lock(&mining_thr_lock);
+			rd_lock(&mining_thr_lock);
 			for (i = 0; i < mining_threads; i++)
 				mining_thr[i]->pause = true;
-			mutex_unlock(&mining_thr_lock);
+			rd_unlock(&mining_thr_lock);
 		} else if (sched_paused && should_run()) {
 			applog(LOG_WARNING, "Restarting execution as per start time %02d:%02d scheduled",
 				schedstart.tm.tm_hour, schedstart.tm.tm_min);
@@ -7734,7 +7950,35 @@ char *curses_input(const char *query)
 }
 #endif
 
-void add_pool_details(struct pool *pool, bool live, char *url, char *user, char *pass)
+static bool pools_active = false;
+
+static void *test_pool_thread(void *arg)
+{
+	struct pool *pool = (struct pool *)arg;
+
+	if (pool_active(pool, false)) {
+		pool_tset(pool, &pool->lagging);
+		pool_tclear(pool, &pool->idle);
+
+		cg_wlock(&control_lock);
+		if (!pools_active) {
+			currentpool = pool;
+			if (pool->pool_no != 0)
+				applog(LOG_NOTICE, "Switching to pool %d %s - first alive pool", pool->pool_no, pool->rpc_url);
+			pools_active = true;
+		}
+		cg_wunlock(&control_lock);
+		pool_resus(pool);
+	} else
+		pool_died(pool);
+
+	return NULL;
+}
+
+/* Always returns true that the pool details were added unless we are not
+ * live, implying this is the only pool being added, so if no pools are
+ * active it returns false. */
+bool add_pool_details(struct pool *pool, bool live, char *url, char *user, char *pass)
 {
 	pool->rpc_url = url;
 	pool->rpc_user = user;
@@ -7744,17 +7988,17 @@ void add_pool_details(struct pool *pool, bool live, char *url, char *user, char
 		quit(1, "Failed to malloc userpass");
 	sprintf(pool->rpc_userpass, "%s:%s", pool->rpc_user, pool->rpc_pass);
 
+	pool->testing = true;
+	pool->idle = true;
 	enable_pool(pool);
 
-	/* Prevent noise on startup */
-	pool->lagging = true;
-
-	/* Test the pool is not idle if we're live running, otherwise
-	 * it will be tested separately */
-	if (live && !pool_active(pool, false)) {
-		gettimeofday(&pool->tv_idle, NULL);
-		pool->idle = true;
+	pthread_create(&pool->test_thread, NULL, test_pool_thread, (void *)pool);
+	if (!live) {
+		pthread_join(pool->test_thread, NULL);
+		pool->testing = false;
+		return pools_active;
 	}
+	return true;
 }
 
 #ifdef HAVE_CURSES
@@ -7794,8 +8038,7 @@ static bool input_pool(bool live)
 		url = httpinput;
 	}
 
-	add_pool_details(pool, live, url, user, pass);
-	ret = true;
+	ret = add_pool_details(pool, live, url, user, pass);
 out:
 	immedok(logwin, false);
 
@@ -7950,9 +8193,9 @@ static int device_line_id_count;
 void register_device(struct cgpu_info *cgpu)
 {
 	cgpu->deven = DEV_ENABLED;
-	mutex_lock(&devices_lock);
+	wr_lock(&devices_lock);
 	devices[cgpu->cgminer_id = cgminer_id_count++] = cgpu;
-	mutex_unlock(&devices_lock);
+	wr_unlock(&devices_lock);
 	if (!cgpu->proc_id)
 		cgpu->device_line_id = device_line_id_count++;
 	mining_threads += cgpu->threads ?: 1;
@@ -7964,6 +8207,9 @@ void register_device(struct cgpu_info *cgpu)
 		gpu_threads += cgpu->threads;
 	}
 #endif
+
+	rwlock_init(&cgpu->qlock);
+	cgpu->queued_work = NULL;
 }
 
 struct _cgpu_devid_counter {
@@ -7999,9 +8245,20 @@ extern void setup_pthread_cancel_workaround();
 extern struct sigaction pcwm_orig_term_handler;
 #endif
 
+static void probe_pools(void)
+{
+	int i;
+
+	for (i = 0; i < total_pools; i++) {
+		struct pool *pool = pools[i];
+
+		pool->testing = true;
+		pthread_create(&pool->test_thread, NULL, test_pool_thread, (void *)pool);
+	}
+}
+
 int main(int argc, char *argv[])
 {
-	bool pools_active = false;
 	struct sigaction handler;
 	struct thr_info *thr;
 	struct block *block;
@@ -8035,17 +8292,16 @@ int main(int argc, char *argv[])
 #endif
 
 	mutex_init(&hash_lock);
-	mutex_init(&qd_lock);
 	mutex_init(&console_lock);
-	mutex_init(&control_lock);
+	cglock_init(&control_lock);
 	mutex_init(&stats_lock);
 	mutex_init(&sharelog_lock);
-	mutex_init(&ch_lock);
+	cglock_init(&ch_lock);
 	mutex_init(&sshare_lock);
 	rwlock_init(&blk_lock);
 	rwlock_init(&netacc_lock);
-	mutex_init(&mining_thr_lock);
-	mutex_init(&devices_lock);
+	rwlock_init(&mining_thr_lock);
+	rwlock_init(&devices_lock);
 
 	mutex_init(&lp_lock);
 	if (unlikely(pthread_cond_init(&lp_cond, NULL)))
@@ -8422,29 +8678,14 @@ int main(int argc, char *argv[])
 
 	applog(LOG_NOTICE, "Probing for an alive pool");
 	do {
-		/* Look for at least one active pool before starting */
-		for (j = 0; j < total_pools; j++) {
-			for (i = 0; i < total_pools; i++) {
-				struct pool *pool  = pools[i];
+		int slept = 0;
 
-				if (pool->prio != j)
-					continue;
-
-				if (pool_active(pool, false)) {
-					pool_tset(pool, &pool->lagging);
-					pool_tclear(pool, &pool->idle);
-					if (!currentpool)
-						currentpool = pool;
-					applog(LOG_INFO, "Pool %d %s active", pool->pool_no, pool->rpc_url);
-					pools_active = true;
-					goto found_active_pool;
-				} else {
-					if (pool == currentpool)
-						currentpool = NULL;
-					applog(LOG_WARNING, "Unable to get work from pool %d %s", pool->pool_no, pool->rpc_url);
-				}
-			}
-		}
+		/* Look for at least one active pool before starting */
+		probe_pools();
+		do {
+			sleep(1);
+			slept++;
+		} while (!pools_active && slept < 60);
 
 		if (!pools_active) {
 			applog(LOG_ERR, "No servers were found that could be used to get work from.");
@@ -8469,7 +8710,6 @@ int main(int argc, char *argv[])
 				quit(0, "No servers could be used! Exiting.");
 		}
 	} while (!pools_active);
-found_active_pool: ;
 
 #ifdef USE_SCRYPT
 	if (detect_algo == 1 && !opt_scrypt) {

miner.h

@@ -293,6 +293,14 @@ struct device_drv {
 	int64_t (*scanhash_queue)(struct thr_info *, struct work **, int64_t);
 #endif
 	int64_t (*scanhash)(struct thr_info *, struct work *, int64_t);
+	int64_t (*scanwork)(struct thr_info *);
+
+	/* Used to extract work from the hash table of queued work and tell
+	 * the main loop that it should not add any further work to the table.
+	 */
+	bool (*queue_full)(struct cgpu_info *);
+	void (*flush_work)(struct cgpu_info *);
+
 	void (*hw_error)(struct thr_info *);
 	void (*thread_shutdown)(struct thr_info *);
 	void (*thread_enable)(struct thr_info *);
@@ -524,6 +532,9 @@ struct cgpu_info {
 	int dev_throttle_count;
 
 	struct cgminer_stats cgminer_stats;
+
+	pthread_rwlock_t qlock;
+	struct work *queued_work;
 };
 
 extern void renumber_cgpu(struct cgpu_info *);
@@ -744,6 +755,65 @@ static inline void rwlock_init(pthread_rwlock_t *lock)
 		quit(1, "Failed to pthread_rwlock_init");
 }
 
+/* cgminer locks, a write biased variant of rwlocks */
+struct cglock {
+	pthread_mutex_t mutex;
+	pthread_rwlock_t rwlock;
+};
+
+typedef struct cglock cglock_t;
+
+static inline void cglock_init(cglock_t *lock)
+{
+	mutex_init(&lock->mutex);
+	rwlock_init(&lock->rwlock);
+}
+
+/* Read lock variant of cglock */
+static inline void cg_rlock(cglock_t *lock)
+{
+	mutex_lock(&lock->mutex);
+	rd_lock(&lock->rwlock);
+	mutex_unlock(&lock->mutex);
+}
+
+/* Intermediate variant of cglock */
+static inline void cg_ilock(cglock_t *lock)
+{
+	mutex_lock(&lock->mutex);
+}
+
+/* Upgrade intermediate variant to a write lock */
+static inline void cg_ulock(cglock_t *lock)
+{
+	wr_lock(&lock->rwlock);
+}
+
+/* Write lock variant of cglock */
+static inline void cg_wlock(cglock_t *lock)
+{
+	mutex_lock(&lock->mutex);
+	wr_lock(&lock->rwlock);
+}
+
+/* Downgrade intermediate variant to a read lock */
+static inline void cg_dlock(cglock_t *lock)
+{
+	rd_lock(&lock->rwlock);
+	mutex_unlock(&lock->mutex);
+}
+
+static inline void cg_runlock(cglock_t *lock)
+{
+	rd_unlock(&lock->rwlock);
+}
+
+static inline void cg_wunlock(cglock_t *lock)
+{
+	wr_unlock(&lock->rwlock);
+	mutex_unlock(&lock->mutex);
+}
+
 struct pool;
 
 extern bool opt_protocol;
@@ -798,11 +868,12 @@ extern int opt_queue;
 extern int opt_scantime;
 extern int opt_expiry;
 
+extern cglock_t control_lock;
 extern pthread_mutex_t hash_lock;
 extern pthread_mutex_t console_lock;
-extern pthread_mutex_t ch_lock;
-extern pthread_mutex_t mining_thr_lock;
-extern pthread_mutex_t devices_lock;
+extern cglock_t ch_lock;
+extern pthread_rwlock_t mining_thr_lock;
+extern pthread_rwlock_t devices_lock;
 
 extern void thread_reportin(struct thr_info *thr);
 extern void thread_reportout(struct thr_info *);
@@ -833,7 +904,7 @@ extern int enabled_pools;
 extern bool detect_stratum(struct pool *pool, char *url);
 extern void print_summary(void);
 extern struct pool *add_pool(void);
-extern void add_pool_details(struct pool *pool, bool live, char *url, char *user, char *pass);
+extern bool add_pool_details(struct pool *pool, bool live, char *url, char *user, char *pass);
 
 #define MAX_GPUDEVICES 16
 
@@ -1026,13 +1097,14 @@ struct pool {
 	char *rpc_proxy;
 
 	pthread_mutex_t pool_lock;
+	cglock_t data_lock;
 
 	struct thread_q *submit_q;
 	struct thread_q *getwork_q;
 
 	pthread_t longpoll_thread;
-	pthread_t submit_thread;
-	pthread_t getwork_thread;
+	pthread_t test_thread;
+	bool testing;
 
 	int curls;
 	pthread_cond_t cr_cond;
@@ -1062,7 +1134,7 @@ struct pool {
 	char *sessionid;
 	bool has_stratum;
 	bool stratum_active;
-	bool stratum_auth;
+	bool stratum_init;
 	bool stratum_notify;
 	struct stratum_work swork;
 	pthread_t stratum_thread;
@@ -1144,7 +1216,12 @@ extern enum test_nonce2_result _test_nonce2(struct work *, uint32_t nonce, bool
 #define test_nonce(work, nonce, checktarget)  (_test_nonce2(work, nonce, checktarget) == TNR_GOOD)
 #define test_nonce2(work, nonce)  (_test_nonce2(work, nonce, true))
 extern void submit_nonce(struct thr_info *thr, struct work *work, uint32_t nonce);
+extern struct work *get_queued(struct cgpu_info *cgpu);
+extern struct work *__find_work_bymidstate(struct work *que, char *midstate, size_t midstatelen, char *data, int offset, size_t datalen);
+struct work *find_queued_work_bymidstate(struct cgpu_info *cgpu, char *midstate, size_t midstatelen, char *data, int offset, size_t datalen);
+extern void work_completed(struct cgpu_info *cgpu, struct work *work);
 extern bool abandon_work(struct work *, struct timeval *work_runtime, uint64_t hashes);
+extern void hash_queued_work(struct thr_info *mythr);
 extern void tailsprintf(char *f, const char *fmt, ...) FORMAT_SYNTAX_CHECK(printf, 2, 3);
 extern void wlog(const char *f, ...) FORMAT_SYNTAX_CHECK(printf, 1, 2);
 extern void wlogprint(const char *f, ...) FORMAT_SYNTAX_CHECK(printf, 1, 2);

ocl.c

@@ -676,8 +676,6 @@ _clState *initCl(unsigned int gpu, char *name, size_t nameSize)
 
 #ifdef USE_SCRYPT
 	if (opt_scrypt) {
-		cl_ulong ma = cgpu->max_alloc, mt;
-
 		if (!cgpu->opt_lg) {
 			applog(LOG_DEBUG, "GPU %d: selecting lookup gap of 2", gpu);
 			cgpu->lookup_gap = 2;
@@ -685,29 +683,18 @@ _clState *initCl(unsigned int gpu, char *name, size_t nameSize)
 			cgpu->lookup_gap = cgpu->opt_lg;
 
 		if (!cgpu->opt_tc) {
-			cgpu->thread_concurrency = ma / 32768 / cgpu->lookup_gap;
+			unsigned int sixtyfours;
+
+			sixtyfours =  cgpu->max_alloc / 131072 / 64 - 1;
+			cgpu->thread_concurrency = sixtyfours * 64;
 			if (cgpu->shaders && cgpu->thread_concurrency > cgpu->shaders) {
 				cgpu->thread_concurrency -= cgpu->thread_concurrency % cgpu->shaders;
 				if (cgpu->thread_concurrency > cgpu->shaders * 5)
 					cgpu->thread_concurrency = cgpu->shaders * 5;
 			}
-				
 			applog(LOG_DEBUG, "GPU %u: selecting thread concurrency of %lu", gpu,  (unsigned long)cgpu->thread_concurrency);
 		} else
 			cgpu->thread_concurrency = cgpu->opt_tc;
-
-		/* If we have memory to spare, try to find a power of 2 value
-		 * >= required amount to map nicely to an intensity */
-		mt = cgpu->thread_concurrency * 32768 * cgpu->lookup_gap;
-		if (ma > mt) {
-			ma = 1;
-			while (ma < mt)
-				ma <<= 1;
-			if (ma < cgpu->max_alloc) {
-				cgpu->max_alloc = ma;
-				applog(LOG_DEBUG, "Max alloc decreased to %lu", (unsigned long)cgpu->max_alloc);
-			}
-		}
 	}
 #endif
 
@@ -1021,8 +1008,7 @@ built:
 		if (bufsize > cgpu->max_alloc) {
 			applog(LOG_WARNING, "Maximum buffer memory device %d supports says %lu", gpu, (unsigned long)cgpu->max_alloc);
 			applog(LOG_WARNING, "Your scrypt settings come to %lu", (unsigned long)bufsize);
-		} else
-			bufsize = cgpu->max_alloc;
+		}
 		applog(LOG_DEBUG, "Creating scrypt buffer sized %lu", (unsigned long)bufsize);
 		clState->padbufsize = bufsize;
 

util.c

@@ -234,14 +234,14 @@ out:
 
 static int keep_sockalive(SOCKETTYPE fd)
 {
-	const int tcp_keepidle = 60;
-	const int tcp_keepintvl = 60;
+	const int tcp_keepidle = 45;
+	const int tcp_keepintvl = 30;
 	const int keepalive = 1;
 	int ret = 0;
 
 
 #ifndef WIN32
-	const int tcp_keepcnt = 5;
+	const int tcp_keepcnt = 1;
 
 	if (unlikely(setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &keepalive, sizeof(keepalive))))
 		ret = 1;
@@ -1473,7 +1473,7 @@ static bool parse_notify(struct pool *pool, json_t *val)
 		goto out;
 	}
 
-	mutex_lock(&pool->pool_lock);
+	cg_wlock(&pool->data_lock);
 	free(pool->swork.job_id);
 	free(pool->swork.prev_hash);
 	free(pool->swork.coinbase1);
@@ -1513,7 +1513,7 @@ static bool parse_notify(struct pool *pool, json_t *val)
 	/* workpadding */	 96;
 	pool->swork.header_len = pool->swork.header_len * 2 + 1;
 	align_len(&pool->swork.header_len);
-	mutex_unlock(&pool->pool_lock);
+	cg_wunlock(&pool->data_lock);
 
 	applog(LOG_DEBUG, "Received stratum notify from pool %u with job_id=%s",
 	       pool->pool_no, job_id);
@@ -1535,7 +1535,7 @@ static bool parse_notify(struct pool *pool, json_t *val)
 	total_getworks++;
 
 	if ((merkles && (!pool->swork.transparency_probed || rand() <= RAND_MAX / (opt_skip_checks + 1))) || pool->swork.transparency_time != (time_t)-1)
-		if (pool->stratum_auth)
+		if (pool->stratum_init)
 			stratum_probe_transparency(pool);
 
 	ret = true;
@@ -1551,9 +1551,9 @@ static bool parse_diff(struct pool *pool, json_t *val)
 	if (diff == 0)
 		return false;
 
-	mutex_lock(&pool->pool_lock);
+	cg_wlock(&pool->data_lock);
 	pool->swork.diff = diff;
-	mutex_unlock(&pool->pool_lock);
+	cg_wunlock(&pool->data_lock);
 
 	applog(LOG_DEBUG, "Pool %d stratum bdifficulty set to %f", pool->pool_no, diff);
 
@@ -1607,9 +1607,10 @@ static bool send_version(struct pool *pool, json_t *val)
 
 static bool stratum_show_message(struct pool *pool, json_t *val, json_t *params)
 {
+	char *msg;
 	char s[RBUFSIZE], *idstr;
 	json_t *id = json_object_get(val, "id");
-	char *msg = json_array_string(params, 0);
+	msg = json_array_string(params, 0);
 	
 	if (likely(msg))
 	{
@@ -1746,7 +1747,7 @@ bool auth_stratum(struct pool *pool)
 			ss = json_dumps(err_val, JSON_INDENT(3));
 		else
 			ss = strdup("(unknown reason)");
-		applog(LOG_WARNING, "JSON stratum auth failed: %s", ss);
+		applog(LOG_WARNING, "pool %d JSON stratum auth failed: %s", pool->pool_no, ss);
 		free(ss);
 
 		goto out;
@@ -1755,7 +1756,6 @@ bool auth_stratum(struct pool *pool)
 	ret = true;
 	applog(LOG_INFO, "Stratum authorisation success for pool %d", pool->pool_no);
 	pool->probed = true;
-	pool->stratum_auth = true;
 	successful_connect = true;
 out:
 	if (val)
@@ -1785,7 +1785,6 @@ static bool setup_stratum_curl(struct pool *pool)
 	mutex_lock(&pool->stratum_lock);
 	pool->swork.transparency_time = (time_t)-1;
 	pool->stratum_active = false;
-	pool->stratum_auth = false;
 	pool->stratum_notify = false;
 	pool->swork.transparency_probed = false;
 	if (pool->stratum_curl)
@@ -1883,25 +1882,46 @@ out:
 	return ret;
 }
 
+void suspend_stratum(struct pool *pool)
+{
+	clear_sockbuf(pool);
+	applog(LOG_INFO, "Closing socket for stratum pool %d", pool->pool_no);
+	mutex_lock(&pool->stratum_lock);
+	pool->stratum_active = pool->stratum_notify = false;
+	if (pool->stratum_curl) {
+		curl_easy_cleanup(pool->stratum_curl);
+	}
+	pool->stratum_curl = NULL;
+	pool->sock = INVSOCK;
+	mutex_unlock(&pool->stratum_lock);
+}
+
 bool initiate_stratum(struct pool *pool)
 {
+	bool ret = false, recvd = false, noresume = false, sockd = false;
 	char s[RBUFSIZE], *sret = NULL, *nonce1, *sessionid;
-	bool ret = false, recvd = false, noresume = false;
 	json_t *val = NULL, *res_val, *err_val;
 	json_error_t err;
 	int n2size;
 
-	if (!setup_stratum_curl(pool))
+resend:
+	if (!setup_stratum_curl(pool)) {
+		sockd = false;
 		goto out;
+	}
 
-resend:
-	if (!noresume) {
+	sockd = true;
+
+	if (noresume) {
+		/* Get rid of any crap lying around if we're resending */
+		clear_sock(pool);
+		sprintf(s, "{\"id\": %d, \"method\": \"mining.subscribe\", \"params\": []}", swork_id++);
+	} else {
 		if (pool->sessionid)
 			sprintf(s, "{\"id\": %d, \"method\": \"mining.subscribe\", \"params\": [\""PACKAGE"/"VERSION"\", \"%s\"]}", swork_id++, pool->sessionid);
 		else
 			sprintf(s, "{\"id\": %d, \"method\": \"mining.subscribe\", \"params\": [\""PACKAGE"/"VERSION"\"]}", swork_id++);
-	} else
-		sprintf(s, "{\"id\": %d, \"method\": \"mining.subscribe\", \"params\": []}", swork_id++);
+	}
 
 	if (__stratum_send(pool, s, strlen(s)) != SEND_OK) {
 		applog(LOG_DEBUG, "Failed to send s in initiate_stratum");
@@ -1962,14 +1982,14 @@ resend:
 		goto out;
 	}
 
-	mutex_lock(&pool->pool_lock);
+	cg_wlock(&pool->data_lock);
 	free(pool->sessionid);
 	pool->sessionid = sessionid;
 	free(pool->nonce1);
 	pool->nonce1 = nonce1;
 	pool->n1_len = strlen(nonce1) / 2;
 	pool->n2size = n2size;
-	mutex_unlock(&pool->pool_lock);
+	cg_wunlock(&pool->data_lock);
 
 	if (sessionid)
 		applog(LOG_DEBUG, "Pool %d stratum session id: %s", pool->pool_no, pool->sessionid);
@@ -1995,28 +2015,13 @@ out:
 			goto resend;
 		}
 		applog(LOG_DEBUG, "Initiate stratum failed");
-		if (pool->sock != INVSOCK) {
-			shutdown(pool->sock, SHUT_RDWR);
-			pool->sock = INVSOCK;
-		}
+		if (sockd)
+			suspend_stratum(pool);
 	}
 
 	return ret;
 }
 
-void suspend_stratum(struct pool *pool)
-{
-	clear_sockbuf(pool);
-	applog(LOG_INFO, "Closing socket for stratum pool %d", pool->pool_no);
-	mutex_lock(&pool->stratum_lock);
-	pool->stratum_active = pool->stratum_notify = false;
-	pool->stratum_auth = false;
-	curl_easy_cleanup(pool->stratum_curl);
-	pool->stratum_curl = NULL;
-	pool->sock = INVSOCK;
-	mutex_unlock(&pool->stratum_lock);
-}
-
 bool restart_stratum(struct pool *pool)
 {
 	if (pool->stratum_active)